{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T05:58:45Z","timestamp":1774418325726,"version":"3.50.1"},"reference-count":85,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,7,23]],"date-time":"2021-07-23T00:00:00Z","timestamp":1626998400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1812968 and 1813253"],"award-info":[{"award-number":["1812968 and 1813253"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2021,10,31]]},"abstract":"<jats:p>Software engineers are crowdsourcing answers to their everyday challenges on Q&amp;A forums (e.g., Stack Overflow) and more recently in public chat communities such as Slack, IRC, and Gitter. Many software-related chat conversations contain valuable expert knowledge that is useful for both mining to improve programming support tools and for readers who did not participate in the original chat conversations. However, most chat platforms and communities do not contain built-in quality indicators (e.g., accepted answers, vote counts). Therefore, it is difficult to identify conversations that contain useful information for mining or reading, i.e., conversations of post hoc quality. In this article, we investigate automatically detecting developer conversations of post hoc quality from public chat channels. We first describe an analysis of 400 developer conversations that indicate potential characteristics of post hoc quality, followed by a machine learning-based approach for automatically identifying conversations of post hoc quality. Our evaluation of 2,000 annotated Slack conversations in four programming communities (python, clojure, elm, and racket) indicates that our approach can achieve precision of 0.82, recall of 0.90, F-measure of 0.86, and MCC of 0.57. To our knowledge, this is the first automated technique for detecting developer conversations of post hoc quality.<\/jats:p>","DOI":"10.1145\/3450503","type":"journal-article","created":{"date-parts":[[2021,7,23]],"date-time":"2021-07-23T10:58:28Z","timestamp":1627037908000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Automatically Identifying the Quality of Developer Chats for Post Hoc Use"],"prefix":"10.1145","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3057-7807","authenticated-orcid":false,"given":"Preetha","family":"Chatterjee","sequence":"first","affiliation":[{"name":"University of Delaware, USA"}]},{"given":"Kostadin","family":"Damevski","sequence":"additional","affiliation":[{"name":"Virginia Commonwealth University, USA"}]},{"given":"Nicholas A.","family":"Kraft","sequence":"additional","affiliation":[{"name":"UserVoice, USA"}]},{"given":"Lori","family":"Pollock","sequence":"additional","affiliation":[{"name":"University of Delaware, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,7,23]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"smart-words.org. 2020. Retrieved from https:\/\/www.smart-words.org\/abbreviations\/text.html.  smart-words.org. 2020. Retrieved from https:\/\/www.smart-words.org\/abbreviations\/text.html."},{"key":"e_1_2_1_2_1","unstructured":"spaCy. 2020. Retrieved from https:\/\/spacy.io\/.  spaCy. 2020. Retrieved from https:\/\/spacy.io\/."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341557"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 13th International Conference on Mining Software Repositories (MSR\u201916)","author":"Ahasanuzzaman Muhammad"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM\u201917)","author":"Alkadhi R.","year":"2017"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR\u201917)","author":"Alkadhi R.","year":"2017"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201918)","volume":"00","author":"Alkadhi R.","year":"2018"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories (MSR\u201913)","author":"Asaduzzaman Muhammad"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/RSSE.2012.6233404"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2830544.2830547"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Association for Computational Linguistics.","author":"Bird Steven","year":"2002"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories (MSR\u201913)","author":"Bosu Amiangshu"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3316550"},{"key":"e_1_2_1_14_1","volume-title":"A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: A review.Arch. Orofacial Sci. 12, 1","author":"Bujang Mohamad Adam","year":"2017"},{"key":"e_1_2_1_15_1","unstructured":"Jonathan P. Chang Caleb Chiam Liye Fu Andrew Z. Wang Justine Zhang and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for the Analysis of Conversations. Retrieved from https:\/\/arxiv:cs.CL\/2005.04246.  Jonathan P. Chang Caleb Chiam Liye Fu Andrew Z. Wang Justine Zhang and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A Toolkit for the Analysis of Conversations. Retrieved from https:\/\/arxiv:cs.CL\/2005.04246."},{"key":"e_1_2_1_16_1","first-page":"19","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP\u201919)","author":"Jonathan","year":"1865"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 17th International Conference on Mining Software Repositories (MSR\u201920)","author":"Chatterjee P."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 16th International Conference on Mining Software Repositories (MSR\u201919)","author":"Chatterjee P.","year":"2019"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR\u201917)","author":"Chatterjee P.","year":"2017"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering. 338\u2013348","author":"Chen C.","year":"2016"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.48"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.48"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the IEEE\/ACM 12th Working Conference on Mining Software Repositories. 422\u2013425","author":"Chowdhury S. A.","year":"2015"},{"key":"e_1_2_1_24_1","unstructured":"Per Christensson. TechTerms.com. Retrieved from https:\/\/techterms.com\/category\/software.  Per Christensson. TechTerms.com. Retrieved from https:\/\/techterms.com\/category\/software."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0076540"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390415"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2512938.2512954"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568036"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.51"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME\u201917)","author":"Ebert F."},{"key":"e_1_2_1_31_1","first-page":"1","article-title":"An empirical study of developer discussions in the gitter platform","volume":"30","author":"Hassan Safwat","year":"2020","journal-title":"Trans. Softw. Eng. Methodol."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work (GROUP\u201903)","author":"Margaret"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the Association of Computational Linguistics: Human Language Technology. 834\u2013842","author":"Elsner Micha","year":"2008"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551511415584"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0057532"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/2820518.2820573"},{"key":"e_1_2_1_37_1","volume-title":"The Technique of Clear Writing","author":"Gunning R."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201909)","author":"Harper F. Maxwell","year":"1870"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201908)","author":"Harper F. Maxwell"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201909)","author":"Hong Liangjie","year":"1941"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories (MSR). 377\u2013386","author":"Howard M. J.","year":"2013"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14505001052"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14505001105"},{"key":"e_1_2_1_45_1","volume-title":"Koch","author":"Richard Landis J.","year":"1977"},{"key":"e_1_2_1_46_1","unstructured":"Carlene Lebeuf Margaret-Anne D. Storey and Alexey Zagalsky. 2017. How software developers mitigate collaboration friction with chatbots. Retrieved from http:\/\/arxiv.org\/abs\/1702.07011.  Carlene Lebeuf Margaret-Anne D. Storey and Alexey Zagalsky. 2017. How software developers mitigate collaboration friction with chatbots. Retrieved from http:\/\/arxiv.org\/abs\/1702.07011."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187980.2188200"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2869117"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00066"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-4640"},{"key":"e_1_2_1_51_1","first-page":"8","article-title":"SMOG grading\u2014A new readability formula.J","volume":"12","author":"McLaughlin H. G.","year":"1969","journal-title":"Reading"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2948063"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/375360.375365"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE\u201918)","author":"Paikari Elahe","year":"1958"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution. 251\u2013260","author":"Panichella S.","year":"2014"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597077"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution. 541\u2013544","author":"Ponzanelli L.","year":"2014"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116251"},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering. 349\u2013359","author":"Rahman M. M.","year":"2016"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering. 194\u2013203","author":"Rahman M. M.","year":"2014"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the IEEE 15th International Working Conference on Source Code Analysis and Manipulation. 81\u201390","author":"Rahman M. M.","year":"2015"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the Conference on Human Information Interaction and Retrieval (CHIIR\u201917)","author":"Rath Manasa","year":"2017"},{"key":"e_1_2_1_63_1","volume-title":"Case Study Research in Software Engineering: Guidelines and Examples","author":"Runeson Per","edition":"1"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380356"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 6th IEEE International Working Conference on Mining Software Repositories (MSR\u201909)","author":"Shihab Emad","year":"2009"},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance. 147\u2013156","author":"Shihab E.","year":"2009"},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance (ICSM\u201912)","author":"Sillito Jonathan","year":"2012"},{"key":"e_1_2_1_68_1","unstructured":"E. A. Smith and R. J. Senter. 1967. Automated readability index. AMRL TR (May 1967) 1\u201314.  E. A. Smith and R. J. Senter. 1967. Automated readability index. AMRL TR (May 1967) 1\u201314."},{"key":"e_1_2_1_69_1","unstructured":"The Statistics Portal Statista. 2020. Retrieved from https:\/\/www.statista.com\/statistics\/652779\/worldwide-slack-users-total-vs-paid\/.  The Statistics Portal Statista. 2020. Retrieved from https:\/\/www.statista.com\/statistics\/652779\/worldwide-slack-users-total-vs-paid\/."},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the 10th International Conference on Computational Semantics (IWCS\u201913)","author":"Stone Matthew","year":"2013"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593887"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3160588"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering. 44\u201353","author":"Tian Y.","year":"2014"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR-WCRE.2014.6747213"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2013.02.004"},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 10th Working Conference on Mining Software Repositories. 61\u201364","author":"Wang W.","year":"2013"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693113"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236031"},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the IEEE 40th Annual Computer Software and Applications Conference (COMPSAC\u201916)","volume":"1","author":"Xia X.","year":"2016"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2901767"},{"key":"e_1_2_1_81_1","volume-title":"C (May","author":"Yao Yuan","year":"2015"},{"key":"e_1_2_1_82_1","volume-title":"Communications in Global Software Development: An Empirical Study Using GTK+ OSS Repository","author":"Yu Liguo"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14886","author":"Zhang Amy","year":"2017"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-015-1576-4"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPC.2005.859725"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3450503","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3450503","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3450503","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:45Z","timestamp":1750191525000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3450503"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,23]]},"references-count":85,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,10,31]]}},"alternative-id":["10.1145\/3450503"],"URL":"https:\/\/doi.org\/10.1145\/3450503","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,23]]},"assertion":[{"value":"2020-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}