{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T00:43:32Z","timestamp":1773276212383,"version":"3.50.1"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,3,30]],"date-time":"2023-03-30T00:00:00Z","timestamp":1680134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>\n            Pull requests are a key part of the collaborative software development and code review process today. However, pull requests can also slow down the software development process when the reviewer(s) or the author do not actively engage with the pull request. In this work, we design an end-to-end service, Nudge, for accelerating overdue pull requests toward completion by reminding the author or the reviewer(s) to engage with their overdue pull requests. First, we use models based on effort estimation and machine learning to predict the completion time for a given pull request. Second, we use activity detection to filter out pull requests that may be overdue but for which sufficient action is taking place nonetheless. Last, we use actor identification to understand who the blocker of the pull request is and\n            <jats:italic>nudge<\/jats:italic>\n            the appropriate actor (author or reviewer(s)). The\u00a0key novelty of Nudge is that it succeeds in reducing pull request resolution time, while ensuring that developers perceive the notifications sent as useful, at the scale of thousands of repositories. In a randomized trial on 147 repositories in use at Microsoft, Nudge was able to reduce pull request resolution time by 60% for 8,500 pull requests, when compared to overdue pull requests for which Nudge did not send a notification. Furthermore, developers receiving Nudge notifications resolved 73% of these notifications as positive. We\u00a0observed similar results when scaling up the deployment of Nudge to 8,000 repositories at Microsoft, for which Nudge sent 210,000 notifications during a full year. This demonstrates Nudge\u2019s ability to scale to thousands of repositories. Last, our qualitative analysis of a selection of Nudge notifications indicates areas for future research, such as taking dependencies among pull requests and developer availability into account.\n          <\/jats:p>","DOI":"10.1145\/3544791","type":"journal-article","created":{"date-parts":[[2022,6,25]],"date-time":"2022-06-25T09:35:37Z","timestamp":1656149737000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Nudge: Accelerating Overdue Pull Requests toward Completion"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9432-1045","authenticated-orcid":false,"given":"Chandra","family":"Maddila","sequence":"first","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0068-6204","authenticated-orcid":false,"given":"Sai Surya","family":"Upadrasta","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0102-8139","authenticated-orcid":false,"given":"Chetan","family":"Bansal","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1358-4124","authenticated-orcid":false,"given":"Nachiappan","family":"Nagappan","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8495-7939","authenticated-orcid":false,"given":"Georgios","family":"Gousios","sequence":"additional","affiliation":[{"name":"Delft University of Technology, XE Delft, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4850-3312","authenticated-orcid":false,"given":"Arie","family":"van Deursen","sequence":"additional","affiliation":[{"name":"Delft University of Technology, XE Delft, The Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2023,3,30]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Azure DevOps REST API. Retrieved 2020 from https:\/\/docs.microsoft.com\/en-us\/rest\/api\/azure\/devops\/?view=azure-devops-rest-5.0."},{"key":"e_1_3_2_3_2","unstructured":"GitHub. Retrieved 2020 from https:\/\/flow.microsoft.com\/en-us\/."},{"key":"e_1_3_2_4_2","unstructured":"Accessed 2020. GitHub. Retrieved 2020 from https:\/\/flow.microsoft.com\/en-us\/blog\/sending-pull-request-review-reminders-using-ms-flows\/."},{"key":"e_1_3_2_5_2","unstructured":"GitHub. Retrieved 2020 from https:\/\/www.openml.org\/a\/estimation-procedures\/9."},{"key":"e_1_3_2_6_2","unstructured":"GitHub Marketplace. Retrieved 2020 from https:\/\/github.com\/marketplace."},{"key":"e_1_3_2_7_2","unstructured":"Long-Running Branches Considered Harmful. Retrieved from https:\/\/blog.newrelic.com\/culture\/long-running-branches-considered-harmful\/."},{"key":"e_1_3_2_8_2","first-page":"937","volume-title":"Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Asthana Sumit","year":"2019","unstructured":"Sumit Asthana, Rahul Kumar, Ranjita Bhagwan, Christian Bird, Chetan Bansal, Chandra Maddila, Sonu Mehta, and B. Ashok. 2019. Whodo: Automating reviewer suggestions at scale. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 937\u2013945."},{"key":"e_1_3_2_9_2","first-page":"167","volume-title":"Proceedings of the 4th International Conference on Computational Intelligence, Communication Systems and Networks","author":"Attarzadeh Iman","year":"2012","unstructured":"Iman Attarzadeh, Amin Mehranzadeh, and Ali Barati. 2012. Proposing an enhanced artificial neural network prediction model to improve the accuracy in software effort estimation. In Proceedings of the 4th International Conference on Computational Intelligence, Communication Systems and Networks. IEEE, 167\u2013172."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1145\/3377813.3381353","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice","author":"Bansal Chetan","year":"2020","unstructured":"Chetan Bansal, Sundararajan Renganathan, Ashima Asudani, Olivier Midy, and Mathru Janakiraman. 2020. DeCaf: diagnosing and triaging performance issues in large-scale cloud services. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice. ACM, 201\u2013210."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2013.6671287"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9292-6"},{"key":"e_1_3_2_13_2","first-page":"493","volume-title":"Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918)","author":"Bhagwan Ranjita","year":"2018","unstructured":"Ranjita Bhagwan, Rahul Kumar, Chandra Sekhar Maddila, and Adithya Abraham Philip. 2018. Orca: Differential bug localization in large-scale services. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918). USENIX, 493\u2013509."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02249046"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010193"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/302405.302647"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(99)00102-8"},{"key":"e_1_3_2_18_2","volume-title":"Proceedings Workshop on Psychology of Programming Interest Group (PPIG\u201910)","author":"Calikli Gul","year":"2010","unstructured":"Gul Calikli, Berna A. Uzundag, and Ayse Bener. 2010. Confirmation bias in software development and testing: An analysis of the effects of company size, experience and reasoning skills. In Proceedings Workshop on Psychology of Programming Interest Group (PPIG\u201910), Rebecca Yates and Fabian Fagerholm (Eds.)."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799958"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208589"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387478"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106256"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/LifeTech52111.2021.9391828"},{"key":"e_1_3_2_24_2","first-page":"48","volume-title":"Proceedings of the IEEE\/ACM 1st International Workshop on Bots in Software Engineering (BotSE\u201919)","author":"Kumar Rahul","year":"2019","unstructured":"Rahul Kumar, Chetan Bansal, Chandra Maddila, Nitin Sharma, Shawn Martelock, and Ravi Bhargava. 2019. Building sankie: An AI platform for devops. In Proceedings of the IEEE\/ACM 1st International Workshop on Bots in Software Engineering (BotSE\u201919). IEEE, 48\u201353."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.matpr.2021.02.610"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/1370750.1370762"},{"key":"e_1_3_2_27_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 1st International Workshop on Bots in Software Engineering (BotSE\u201919)","author":"Lebeuf Carlene","year":"2019","unstructured":"Carlene Lebeuf, Alexey Zagalsky, Matthieu Foucault, and Margaret-Anne Storey. 2019. Defining and classifying software bots: a faceted taxonomy. In Proceedings of the IEEE\/ACM 1st International Workshop on Bots in Software Engineering (BotSE\u201919). IEEE, 1\u20136."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291003"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.265100500"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3340457"},{"key":"e_1_3_2_31_2","first-page":"435","volume-title":"Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201920)","author":"Mehta Sonu","year":"2020","unstructured":"Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Maddila, B. Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2020. Rex: Preventing bugs and misconfiguration in large services using correlated change analysis. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201920). 435\u2013448."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iot.2020.100213"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.83"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1037\/1089-2680.2.2.175"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007524"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1080\/01639269.2014.964593"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239235.3240504"},{"key":"e_1_3_2_38_2","first-page":"230","volume-title":"Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201919)","author":"Ren Luyao","year":"2019","unstructured":"Luyao Ren, Shurui Zhou, Christian K\u00e4stner, and Andrzej W\u0105sowski. 2019. Identifying redundancies in fork-based development. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201919). IEEE, 230\u2013241."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2695664.2695856"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043957"},{"key":"e_1_3_2_41_2","volume-title":"Dagstuhl Reports","author":"Storey Margaret-Anne","year":"2020","unstructured":"Margaret-Anne Storey, Alexander Serebrenik, Carolyn Penstein Ros\u00e9, Thomas Zimmermann, and James D. Herbsleb. 2020. BOTse: Bots in software engineering (Dagstuhl Seminar 19471). In Dagstuhl Reports, Vol. 9. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.111"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568315"},{"key":"e_1_3_2_44_2","first-page":"357","volume-title":"Proceedings of the IEEE\/ACM 12th Working Conference on Mining Software Repositories","author":"Veen Erik Van Der","year":"2015","unstructured":"Erik Van Der Veen, Georgios Gousios, and Andy Zaidman. 2015. Automatically prioritizing pull requests. In Proceedings of the IEEE\/ACM 12th Working Conference on Mining Software Repositories. IEEE, 357\u2013361."},{"key":"e_1_3_2_45_2","first-page":"1","volume-title":"Proceedings of the 11th Asia-Pacific Symposium on Internetware","author":"Wang Qingye","year":"2019","unstructured":"Qingye Wang, Bowen Xu, Xin Xia, Ting Wang, and Shanping Li. 2019. Duplicate pull request detection: When time matters. In Proceedings of the 11th Asia-Pacific Symposium on Internetware. 1\u201310."},{"key":"e_1_3_2_46_2","first-page":"106408","article-title":"Large-scale intent analysis for identifying large-review-effort code changes","author":"Wang Song","year":"2020","unstructured":"Song Wang, Chetan Bansal, and Nachiappan Nagappan. 2020. Large-scale intent analysis for identifying large-review-effort code changes. Inf. Softw. Technol. (2020), 106408.","journal-title":"Inf. Softw. Technol."},{"key":"e_1_3_2_47_2","first-page":"46","volume-title":"Proceedings of the 15th International Conference on Predictive Models and Data Analytics in Software Engineering","author":"Wang Song","year":"2019","unstructured":"Song Wang, Chetan Bansal, Nachiappan Nagappan, and Adithya Abraham Philip. 2019. Leveraging change intents for characterizing and identifying large-review-effort changes. In Proceedings of the 15th International Conference on Predictive Models and Data Analytics in Software Engineering. 46\u201355."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE.2019.00015"},{"key":"e_1_3_2_49_2","first-page":"367","volume-title":"Proceedings of the 12th Working Conference on Mining Software Repositories (MSR\u201915)","author":"Yu Yue","year":"2015","unstructured":"Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. Wait for It: Determinants of Pull Request Evaluation Latency on GitHub. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR\u201915). IEEE Press, Piscataway, NJ, 367\u2013371. http:\/\/dl.acm.org\/citation.cfm?id=2820518.2820564."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.01.004"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3544791","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3544791","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:25Z","timestamp":1750178785000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3544791"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,30]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3544791"],"URL":"https:\/\/doi.org\/10.1145\/3544791","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,30]]},"assertion":[{"value":"2021-03-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-05-20","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}