{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T15:53:22Z","timestamp":1775145202341,"version":"3.50.1"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,2,2]],"date-time":"2018-02-02T00:00:00Z","timestamp":1517529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2018,5,31]]},"abstract":"<jats:p>\n            DevOps is a new approach to drive a seamless Application (App) cycle from development to delivery. As a critical part to promote the successful implementation of DevOps, testing can significantly improve team productivity and reliably deliver user experience. However, it is difficult to use traditional testing to cover diverse mobile phones, network environments, operating systems, and so on. Hence, many large companies crowdsource their App testing tasks to workers from open platforms. In crowdsourced testing, test reports submitted by workers may be highly redundant, and their quality may vary sharply. Meanwhile, multi-bug test reports may be submitted, and their root causes are hard to diagnose. Hence, it is a time-consuming and tedious task for developers to manually inspect these test reports. To help developers address the above challenges, we issue the new problem of Fuzzy Clustering Test Reports (FULTER). Aiming to resolve FULTER, a series of barriers need to be overcome. In this study, we propose a new framework named Test Report Fuzzy Clustering Framework (TERFUR) by aggregating redundant and multi-bug test reports into clusters to reduce the number of inspected test reports. First, we construct a filter to remove invalid test reports to break through the\n            <jats:italic>invalid barrier<\/jats:italic>\n            . Then, a preprocessor is built to enhance the descriptions of short test reports to break through the\n            <jats:italic>uneven barrier<\/jats:italic>\n            . Last, a two-phase merging algorithm is proposed to partition redundant and multi-bug test reports into clusters that can break through the\n            <jats:italic>multi-bug barrier<\/jats:italic>\n            . Experimental results over 1,728 test reports from five industrial Apps show that TERFUR can cluster test reports by up to 78.15% in terms of\n            <jats:italic>AverageP<\/jats:italic>\n            , 78.41% in terms of\n            <jats:italic>AverageR<\/jats:italic>\n            , and 75.82% in terms of\n            <jats:italic>AverageF1<\/jats:italic>\n            and outperform comparative methods by up to 31.69%, 33.06%, and 24.55%, respectively. In addition, the effectiveness of TERFUR is validated in prioritizing test reports for manual inspection.\n          <\/jats:p>","DOI":"10.1145\/3106164","type":"journal-article","created":{"date-parts":[[2018,2,6]],"date-time":"2018-02-06T18:13:28Z","timestamp":1517940808000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":51,"title":["Fuzzy Clustering of Crowdsourced Test Reports for Apps"],"prefix":"10.1145","volume":"18","author":[{"given":"He","family":"Jiang","sequence":"first","affiliation":[{"name":"Dalian University of Technology, Dalian, Liaoning Province, China"}]},{"given":"Xin","family":"Chen","sequence":"additional","affiliation":[{"name":"Dalian University of Technology, Dalian, Liaoning Province, China"}]},{"given":"Tieke","family":"He","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, Jiangsu Province, China"}]},{"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, Jiangsu Province, China"}]},{"given":"Xiaochen","family":"Li","sequence":"additional","affiliation":[{"name":"Dalian University of Technology, Dalian, Liaoning Province, China"}]}],"member":"320","published-online":{"date-parts":[[2018,2,2]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC\u201915)","author":"Balalaie Armin","year":"2015","unstructured":"Armin Balalaie , Abbas Heydarnoori , and Pooyan Jamshidi . 2015 . Migrating to cloud-native architectures using microservices: An experience report . In Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC\u201915) . Springer, 201--215. Armin Balalaie, Abbas Heydarnoori, and Pooyan Jamshidi. 2015. Migrating to cloud-native architectures using microservices: An experience report. In Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC\u201915). Springer, 201--215."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/91.531780"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML\u201908)","author":"Bartz Kevin","year":"2008","unstructured":"Kevin Bartz , Jack W. Stokes , John C. Platt , Ryan Kivett , David Grant , Silviu Calinoiu , and Gretchen Loihle . 2008 . Finding similar failures using callstack similarity . In Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML\u201908) . USENIX Association, San Diego, CA. Kevin Bartz, Jack W. Stokes, John C. Platt, Ryan Kivett, David Grant, Silviu Calinoiu, and Gretchen Loihle. 2008. Finding similar failures using callstack similarity. In Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML\u201908). USENIX Association, San Diego, CA."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Pavel Berkhin. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data\u2014Recent Advances in Clustering. 25--71.  Pavel Berkhin. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data\u2014Recent Advances in Clustering. 25--71.","DOI":"10.1007\/3-540-28349-8_2"},{"key":"e_1_2_1_5_1","volume-title":"Pattern Recognition with Fuzzy Objective Function Algorithms","author":"Bezdek James C.","unstructured":"James C. Bezdek . 2013. Pattern Recognition with Fuzzy Objective Function Algorithms . Springer Science 8 Business Media. James C. Bezdek. 2013. Pattern Recognition with Fuzzy Objective Function Algorithms. Springer Science 8 Business Media."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/WI-IAT.2009.104"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR.2010.52"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591153"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TFUZZ.2015.2403878"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.50"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337364"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.44"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786862"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970367"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/2820116.2820123"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2004.58"},{"key":"e_1_2_1_17_1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","unstructured":"Jeff Howe . 2006 . The rise of crowdsourcing . Wired Mag. 14 , 6 (2006), 1 -- 4 . Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.","journal-title":"Wired Mag."},{"key":"e_1_2_1_18_1","volume-title":"DevOps for Developers","author":"Httermann Michael","unstructured":"Michael Httermann . 2012. DevOps for Developers . Apress . Michael Httermann. 2012. DevOps for Developers. Apress."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/331499.331504"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.122"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2011.5958261"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPRIME.2013.6496451"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2002.1007544"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/FUZZY.1999.790086"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2009.23"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2007.08.001"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2007.190740"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/646713.703485"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504901134"},{"key":"e_1_2_1_30_1","volume-title":"Manning and Hinrich Sch\u00fctze","author":"Christopher","year":"2001","unstructured":"Christopher D. Manning and Hinrich Sch\u00fctze . 2001 . Foundations of Statistical Natural Language Processing. MIT Press , Cambridge. Christopher D. Manning and Hinrich Sch\u00fctze. 2001. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.09.015"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the International Conference on Computational Intelligence. IASTED\/ACTA Press","author":"Massey Louis","year":"2005","unstructured":"Louis Massey . 2005 . Evaluating and comparing text clustering results . In Proceedings of the International Conference on Computational Intelligence. IASTED\/ACTA Press , Alberta, Canada, 85--90. Louis Massey. 2005. Evaluating and comparing text clustering results. In Proceedings of the International Conference on Computational Intelligence. IASTED\/ACTA Press, Alberta, Canada, 85--90."},{"key":"e_1_2_1_33_1","volume-title":"Enhancing the Power of the Internet","author":"Mendes M. E. S.","unstructured":"M. E. S. Mendes and Lionel Sacks . 2004. Dynamic knowledge representation for E-learning applications . In Enhancing the Power of the Internet . Springer , 259--282. M. E. S. Mendes and Lionel Sacks. 2004. Dynamic knowledge representation for E-learning applications. In Enhancing the Power of the Internet. Springer, 259--282."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW.2007.4401026"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31753-8_52"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2351676.2351687"},{"key":"e_1_2_1_37_1","first-page":"1529","article-title":"Fuzzy clustering in web text mining and its application in IEEE abstract classification","volume":"2","author":"Papalkar Rahul R.","year":"2013","unstructured":"Rahul R. Papalkar and G. Chandel . 2013 . Fuzzy clustering in web text mining and its application in IEEE abstract classification . Int. J. Comput. Sci. Manage. Rese. 2 , 2 (2013), 1529 -- 1533 . Rahul R. Papalkar and G. Chandel. 2013. Fuzzy clustering in web text mining and its application in IEEE abstract classification. Int. J. Comput. Sci. Manage. Rese. 2, 2 (2013), 1529--1533.","journal-title":"Int. J. Comput. Sci. Manage. Rese."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.13"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/776816.776872"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC\u201904)","author":"M. E.","unstructured":"M. E. S. Mendes Rodrigues and L. Sacks. 2004. A scalable hierarchical fuzzy clustering algorithm for text mining . In Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC\u201904) . IEEE Computer Society, 269--274. M. E. S. Mendes Rodrigues and L. Sacks. 2004. A scalable hierarchical fuzzy clustering algorithm for text mining. In Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC\u201904). IEEE Computer Society, 269--274."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2904354.2904372"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9236(99)00037-8"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.32"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/2818754.2818836"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.205"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2011.6100061"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806811"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2008.2009463"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962584"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970300"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368151"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2015.23"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC\u201910)","author":"Win Thaung Thaung","year":"2010","unstructured":"Thaung Thaung Win and LinMon. 2010 . Document clustering by fuzzy C-mean algorithm . In Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC\u201910) . IEEE Computer Society, 239--242. Thaung Thaung Win and LinMon. 2010. Document clustering by fuzzy C-mean algorithm. In Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC\u201910). IEEE Computer Society, 239--242."},{"key":"e_1_2_1_54_1","volume-title":"25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u20192016)","author":"Xu Jinglin","year":"2016","unstructured":"Jinglin Xu , Junwei Han , Kai Xiong , and Feiping Nie . 2016 . Robust and sparse fuzzy K-means clustering . In 25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u20192016) . IJCAI\/AAAI Press, New York, 2224--2230. Jinglin Xu, Junwei Han, Kai Xiong, and Feiping Nie. 2016. Robust and sparse fuzzy K-means clustering. In 25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u20192016). IJCAI\/AAAI Press, New York, 2224--2230."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2324590"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE\u201910)","author":"Xuan Jifeng","year":"2010","unstructured":"Jifeng Xuan , He Jiang , Zhilei Ren , Jun Yan , and Zhongxuan Luo . 2010 . Automatic bug triage using semi-supervised text classification . In Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE\u201910) . Knowledge Systems Institute Graduate School, Zurich, Switzerland, 209--214. Jifeng Xuan, He Jiang, Zhilei Ren, Jun Yan, and Zhongxuan Luo. 2010. Automatic bug triage using semi-supervised text classification. In Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE\u201910). Knowledge Systems Institute Graduate School, Zurich, Switzerland, 209--214."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337227"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/SWS.2010.5607428"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2010.01.011"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3106164","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3106164","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:04Z","timestamp":1750217404000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3106164"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,2]]},"references-count":59,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,5,31]]}},"alternative-id":["10.1145\/3106164"],"URL":"https:\/\/doi.org\/10.1145\/3106164","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,2]]},"assertion":[{"value":"2016-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-02-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}