{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:24:20Z","timestamp":1750220660439,"version":"3.41.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2020,9,11]],"date-time":"2020-09-11T00:00:00Z","timestamp":1599782400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["FT140101247"],"award-info":[{"award-number":["FT140101247"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Discovery Project","award":["DP200102298"],"award-info":[{"award-number":["DP200102298"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2020,12,31]]},"abstract":"<jats:p>\n            Truth discovery has been widely studied in recent years as a fundamental means for resolving the conflicts in multi-source data. Although many truth discovery methods have been proposed based on different considerations and intuitions, investigations show that no single method consistently outperforms the others. To select the right truth discovery method for a specific application scenario, it becomes essential to evaluate and compare the performance of different methods. A drawback of current research efforts is that they commonly assume the availability of certain ground truth for the evaluation of methods. However, the ground truth may be very limited or even impossible to obtain, rendering the evaluation biased. In this article, we present\n            <jats:italic>CompTruthHyp<\/jats:italic>\n            , a generic approach for comparing the performance of truth discovery methods without using ground truth. In particular, our approach calculates the probability of observations in a dataset based on the output of different methods. The probability is then ranked to reflect the performance of these methods. We review and compare 12 representative truth discovery methods and consider both single-valued and multi-valued objects. The empirical studies on both real-world and synthetic datasets demonstrate the effectiveness of our approach for comparing truth discovery methods.\n          <\/jats:p>","DOI":"10.1145\/3411749","type":"journal-article","created":{"date-parts":[[2020,9,12]],"date-time":"2020-09-12T19:46:24Z","timestamp":1599939984000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["From Appearance to Essence"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1527-9270","authenticated-orcid":false,"given":"Xiu Susie","family":"Fang","sequence":"first","affiliation":[{"name":"Donghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Quan Z.","family":"Sheng","sequence":"additional","affiliation":[{"name":"Macquarie University, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xianzhi","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Technology Sydney, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei Emma","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Adelaide, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3383-551X","authenticated-orcid":false,"given":"Anne H. H.","family":"Ngu","sequence":"additional","affiliation":[{"name":"Texas State University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jian","family":"Yang","sequence":"additional","affiliation":[{"name":"Macquarie University, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,9,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2847252"},{"volume-title":"Truth Discovery","author":"Berti-\u00c9quille Laure","key":"e_1_2_1_2_1","unstructured":"Laure Berti-\u00c9quille . 2019. Truth Discovery . Springer International Publishing , Cham , 1--8. Laure Berti-\u00c9quille. 2019. Truth Discovery. Springer International Publishing, Cham, 1--8."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1951365.1951414"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1921008"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687690"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687691"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623623"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732951.2732962"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535568.2448938"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-32281-5_1"},{"volume-title":"Search of Elegance in the Theory and Practice of Computation: Essays Dedicated to Peter Buneman","author":"Fan Wenfei","key":"e_1_2_1_12_1","unstructured":"Wenfei Fan , Floris Geerts , Shuai Ma , Nan Tang , and Wenyuan Yu. 2013. Data quality problems beyond consistency and duduplication . In Search of Elegance in the Theory and Practice of Computation: Essays Dedicated to Peter Buneman . Springer Berlin Heidelberg , 237--249. Wenfei Fan, Floris Geerts, Shuai Ma, Nan Tang, and Wenyuan Yu. 2013. Data quality problems beyond consistency and duduplication. In Search of Elegance in the Theory and Practice of Computation: Essays Dedicated to Peter Buneman. Springer Berlin Heidelberg, 237--249."},{"volume-title":"Proceedings of the 36th International Conference on Conceptual Modeling (ER\u201917)","author":"Fang Xiu Susie","key":"e_1_2_1_13_1","unstructured":"Xiu Susie Fang , Quan Z. Sheng , Xianzhi Wang , Mahmoud Barhamgi , Lina Yao , and Anne H. H. Ngu . 2017. SourceVote: Fusing multi-valued data via inter-source agreements . In Proceedings of the 36th International Conference on Conceptual Modeling (ER\u201917) . 164--172. Xiu Susie Fang, Quan Z. Sheng, Xianzhi Wang, Mahmoud Barhamgi, Lina Yao, and Anne H. H. Ngu. 2017. SourceVote: Fusing multi-valued data via inter-source agreements. In Proceedings of the 36th International Conference on Conceptual Modeling (ER\u201917). 164--172."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-018-0629-3"},{"volume-title":"Proceedings of the International World Wide Web Conference (WWW\u201917)","author":"Fang Xiu Susie","key":"e_1_2_1_15_1","unstructured":"Xiu Susie Fang , Quan Z. Sheng , Xianzhi Wang , and Anne H. H. Ngu . 2017. Value veracity estimation for multi-truth objects via a graph-based approach . In Proceedings of the International World Wide Web Conference (WWW\u201917) . 777--778. Xiu Susie Fang, Quan Z. Sheng, Xianzhi Wang, and Anne H. H. Ngu. 2017. Value veracity estimation for multi-truth objects via a graph-based approach. In Proceedings of the International World Wide Web Conference (WWW\u201917). 777--778."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718504"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772730"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963192.1963220"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/324133.324140"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735505"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610509"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535568.2448943"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113275"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897350.2897352"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219998"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3187009.3177739"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402707.3402731"},{"volume-title":"Proceedings of the IEEE 14th International Conference on Mobile Ad Hoc and Sensor Systems (MASS\u201917)","author":"Marshall J.","key":"e_1_2_1_28_1","unstructured":"J. Marshall , A. Argueta , and D. Wang . 2017. A neural network approach for truth discovery in social sensing . In Proceedings of the IEEE 14th International Conference on Mobile Ad Hoc and Sensor Systems (MASS\u201917) . 343--347. J. Marshall, A. Argueta, and D. Wang. 2017. A neural network approach for truth discovery in social sensing. In Proceedings of the IEEE 14th International Conference on Mobile Ad Hoc and Sensor Systems (MASS\u201917). 343--347."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3277505"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Army Science Conference.","author":"Pasternack Jeff","year":"2010","unstructured":"Jeff Pasternack and Dan Roth . 2010 . Comprehensive trust metrics for information networks . In Proceedings of the Army Science Conference. Jeff Pasternack and Dan Roth. 2010. Comprehensive trust metrics for information networks. In Proceedings of the Army Science Conference."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the International Conference on Computational Linguistics (COLING\u201910)","author":"Pasternack Jeff","year":"2010","unstructured":"Jeff Pasternack and Dan Roth . 2010 . Knowing what to believe (when you already know something) . In Proceedings of the International Conference on Computational Linguistics (COLING\u201910) . 877--885. Jeff Pasternack and Dan Roth. 2010. Knowing what to believe (when you already know something). In Proceedings of the International Conference on Computational Linguistics (COLING\u201910). 877--885."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI\u201911)","author":"Pasternack Jeff","year":"2011","unstructured":"Jeff Pasternack and Dan Roth . 2011 . Making better informed trust decisions with generalized fact-finding . In Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI\u201911) . 2324--2329. Jeff Pasternack and Dan Roth. 2011. Making better informed trust decisions with generalized fact-finding. In Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI\u201911). 2324--2329."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488476"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2593674"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3055133"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610504"},{"key":"e_1_2_1_37_1","volume-title":"Truth discovery algorithms: An experimental evaluation. Arxiv Preprint Arxiv:1409.6428","author":"Waguih Dalia Attia","year":"2014","unstructured":"Dalia Attia Waguih and Laure Berti-Equille . 2014. Truth discovery algorithms: An experimental evaluation. Arxiv Preprint Arxiv:1409.6428 ( 2014 ). Dalia Attia Waguih and Laure Berti-Equille. 2014. Truth discovery algorithms: An experimental evaluation. Arxiv Preprint Arxiv:1409.6428 (2014)."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939837"},{"volume-title":"Proceedings of the of the International Conference on Information Processing in Sensor Networks (IPSN\u201914)","author":"Wang D.","key":"e_1_2_1_39_1","unstructured":"D. Wang , M. T. Amin , S. Li , T. Abdelzaher , L. Kaplan , S. Gu , C. Pan , H. Liu , C. C. Aggarwal , R. Ganti , X. Wang , P. Mohapatra , B. Szymanski , and H. Le . 2014. Using humans as sensors: An estimation-theoretic perspective . In Proceedings of the of the International Conference on Information Processing in Sensor Networks (IPSN\u201914) . 35--46. D. Wang, M. T. Amin, S. Li, T. Abdelzaher, L. Kaplan, S. Gu, C. Pan, H. Liu, C. C. Aggarwal, R. Ganti, X. Wang, P. Mohapatra, B. Szymanski, and H. Le. 2014. Using humans as sensors: An estimation-theoretic perspective. In Proceedings of the of the International Conference on Information Processing in Sensor Networks (IPSN\u201914). 35--46."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806443"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983791"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983767"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939831"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2837026"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2007.190745"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963405.1963439"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the International Conference on Computational Linguistics (COLING\u201914)","author":"Yu Dian","year":"2014","unstructured":"Dian Yu , Hongzhao Huang , Taylor Cassidy , Heng Ji , Chi Wang , Shi Zhi , Jiawei Han , Clare Voss , and Malik Magdon-Ismail . 2014 . The wisdom of minority: Unsupervised slot filling validation based on multi-dimensional truth-finding . In Proceedings of the International Conference on Computational Linguistics (COLING\u201914) . 1567--1578. Dian Yu, Hongzhao Huang, Taylor Cassidy, Heng Ji, Chi Wang, Shi Zhi, Jiawei Han, Clare Voss, and Malik Magdon-Ismail. 2014. The wisdom of minority: Unsupervised slot filling validation based on multi-dimensional truth-finding. In Proceedings of the International Conference on Computational Linguistics (COLING\u201914). 1567--1578."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983785"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the International Workshop on Quality in DataBases (QDB\u201912)","author":"Zhao Bo","year":"2012","unstructured":"Bo Zhao and Jiawei Han . 2012 . A probabilistic model for estimating real-valued truth from conflicting sources . In Proceedings of the International Workshop on Quality in DataBases (QDB\u201912) coheld with VLDB. Bo Zhao and Jiawei Han. 2012. A probabilistic model for estimating real-valued truth from conflicting sources. In Proceedings of the International Workshop on Quality in DataBases (QDB\u201912) coheld with VLDB."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.14778\/2168651.2168656"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2018.00097"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783339"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411749","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411749","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:39Z","timestamp":1750197759000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411749"}},"subtitle":["Comparing Truth Discovery Methods without Using Ground Truth"],"short-title":[],"issued":{"date-parts":[[2020,9,11]]},"references-count":53,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,12,31]]}},"alternative-id":["10.1145\/3411749"],"URL":"https:\/\/doi.org\/10.1145\/3411749","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2020,9,11]]},"assertion":[{"value":"2020-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-09-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}