{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:31:20Z","timestamp":1750221080418,"version":"3.41.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,1,4]],"date-time":"2019-01-04T00:00:00Z","timestamp":1546560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Mexican National Council for Science and Technology"},{"name":"UK EPSRC"},{"name":"VADA Programme"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2019,3,31]]},"abstract":"<jats:p>A multicriteria data source selection (MCSS) scenario identifies, from a set of candidate data sources, the subset that best meets users\u2019 needs. These needs are expressed using several criteria, which are used to evaluate the candidate data sources. An MCSS problem can be solved using multidimensional optimization techniques that trade off the different objectives. Sometimes one may have uncertain knowledge regarding how well the candidate data sources meet the criteria. In order to overcome this uncertainty, one may rely on end-users or crowds to annotate the data items produced by the sources in relation to the selection criteria. In this article, a proposed Targeted Feedback Collection (TFC) approach is introduced that aims to identify those data items on which feedback should be collected, thereby providing evidence on how the sources satisfy the required criteria. The proposed TFC targets feedback by considering the confidence intervals around the estimated criteria values, with a view to increasing the confidence in the estimates that are most relevant to the multidimensional optimization. Variants of the proposed TFC approach have been developed for use where feedback is expected to be reliable (e.g., where it is provided by trusted experts) and where feedback is expected to be unreliable (e.g., from crowd workers). Both variants have been evaluated, and positive results are reported against other approaches to feedback collection, including active learning, in experiments that involve real-world datasets and crowdsourcing.<\/jats:p>","DOI":"10.1145\/3284934","type":"journal-article","created":{"date-parts":[[2019,1,7]],"date-time":"2019-01-07T13:42:28Z","timestamp":1546868548000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Crowdsourced Targeted Feedback Collection for Multicriteria Data Source Selection"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0377-6421","authenticated-orcid":false,"given":"Julio C\u00e9sar Cort\u00e9s","family":"R\u00edos","sequence":"first","affiliation":[{"name":"University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norman W.","family":"Paton","sequence":"additional","affiliation":[{"name":"University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alvaro A. A.","family":"Fernandes","sequence":"additional","affiliation":[{"name":"University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edward","family":"Abel","sequence":"additional","affiliation":[{"name":"University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John A.","family":"Keane","sequence":"additional","affiliation":[{"name":"University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,1,4]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Embury","author":"Abel Edward","year":"2018","unstructured":"Edward Abel , Keane John , Norman W. Paton , Fernandes Alvaro A. A. , Martin Koehler , Nikolaos Konstantinou , Nurzety Bintiahmadazuan , and Suzanne M . Embury . 2018 . User driven multi-criteria source selection. Information Sciences 430--431 (2018), 179--199. Edward Abel, Keane John, Norman W. Paton, Fernandes Alvaro A. A., Martin Koehler, Nikolaos Konstantinou, Nurzety Bintiahmadazuan, and Suzanne M. Embury. 2018. User driven multi-criteria source selection. Information Sciences 430--431 (2018), 179--199."},{"volume-title":"New Concepts and Applications in Soft Computing","author":"Barbu Tudor","key":"e_1_2_1_2_1","unstructured":"Tudor Barbu , Mihaela Costin , and Adrian Ciobanu . 2013. Color-based image retrieval approaches using a relevance feedback scheme . In New Concepts and Applications in Soft Computing . Springer-Verlag , Berlin , 47--55. Tudor Barbu, Mihaela Costin, and Adrian Ciobanu. 2013. Color-based image retrieval approaches using a relevance feedback scheme. In New Concepts and Applications in Soft Computing. Springer-Verlag, Berlin, 47--55."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2013.01.006"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187971"},{"volume-title":"Principles of Statistics","author":"Bulmer Michael G.","key":"e_1_2_1_5_1","unstructured":"Michael G. Bulmer . 1979. Principles of Statistics . Dover Publications , New York . Michael G. Bulmer. 1979. Principles of Statistics. Dover Publications, New York."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00039"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-017-1057-x"},{"key":"e_1_2_1_8_1","first-page":"8","article-title":"Wrapper generation supervised by a noisy crowd","volume":"1","author":"Crescenzi Valter","year":"2013","unstructured":"Valter Crescenzi , Paolo Merialdo , and Disheng Qiu . 2013 . Wrapper generation supervised by a noisy crowd . DBCrowd 1 , 1 (2013), 8 -- 13 . Valter Crescenzi, Paolo Merialdo, and Disheng Qiu. 2013. Wrapper generation supervised by a noisy crowd. DBCrowd 1, 1 (2013), 8--13.","journal-title":"DBCrowd"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10619-014-7163-9"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster Arthur P.","year":"1977","unstructured":"Arthur P. Dempster , Nan M. Laird , and Donald B. Rubin . 1977 . Maximum likelihood from incomplete data via the EM algorithm . Journal of the Royal Statistical Society 39 , 1 (1977), 1 -- 38 . Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39, 1 (1977), 1--38.","journal-title":"Journal of the Royal Statistical Society"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535568.2448938"},{"volume-title":"The Survey Handbook","author":"Fink Arlene","key":"e_1_2_1_13_1","unstructured":"Arlene Fink . 2002. The Survey Handbook . SAGE Publications, University of California at Los Angeles, The Langley Research Institute . Arlene Fink. 2002. The Survey Handbook. SAGE Publications, University of California at Los Angeles, The Langley Research Institute."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1972.1054863"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISM.2014.22"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2588576"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3012003"},{"volume-title":"25th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201918)","author":"Grano Giovanni","key":"e_1_2_1_19_1","unstructured":"Giovanni Grano , Adelina Ciurumelea , Sebastiano Panichella , Fabio Palomba , and Harald C. Gall . 2018. Exploring the integration of user feedback in automated testing of Android applications . In 25th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201918) . Campobasso, Italy, 72--83. Giovanni Grano, Adelina Ciurumelea, Sebastiano Panichella, Fabio Palomba, and Harald C. Gall. 2018. Exploring the integration of user feedback in automated testing of Android applications. In 25th International Conference on Software Analysis, Evolution and Reengineering (SANER\u201918). Campobasso, Italy, 72--83."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903730"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350276"},{"key":"e_1_2_1_22_1","first-page":"1953","article-title":"Iterative learning for reliable crowdsourcing systems","volume":"24","author":"Karger David R.","year":"2011","unstructured":"David R. Karger , Sewoong Oh , and Devavrat Shah . 2011 . Iterative learning for reliable crowdsourcing systems . Advances in Neural Information Processing Systems 24 (2011), 1953 -- 1961 . David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. Advances in Neural Information Processing Systems 24 (2011), 1953--1961.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_23_1","first-page":"1","article-title":"Overlapping confidence intervals and statistical significance","volume":"73","author":"Knezevic Andrea","year":"2008","unstructured":"Andrea Knezevic . 2008 . Overlapping confidence intervals and statistical significance . Cornell University, Statistical Consulting Unit 73 (2008), 1 . Andrea Knezevic. 2008. Overlapping confidence intervals and statistical significance. Cornell University, Statistical Consulting Unit 73 (2008), 1.","journal-title":"Cornell University, Statistical Consulting Unit"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-3012"},{"volume-title":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201994)","author":"David","key":"e_1_2_1_25_1","unstructured":"David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers . In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201994) . Springer-Verlag New York,New York, 3--12. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201994). Springer-Verlag New York,New York, 3--12."},{"key":"e_1_2_1_26_1","first-page":"136","article-title":"When 100% really isn\u2019t 100%: Improving the accuracy of small--sample estimates of completion rates","volume":"3","author":"Lewis James R.","year":"2006","unstructured":"James R. Lewis and Jeff Sauro . 2006 . When 100% really isn\u2019t 100%: Improving the accuracy of small--sample estimates of completion rates . Journal of Usability Studies 3 , 1 (2006), 136 -- 150 . James R. Lewis and Jeff Sauro. 2006. When 100% really isn\u2019t 100%: Improving the accuracy of small--sample estimates of completion rates. Journal of Usability Studies 3, 1 (2006), 136--150.","journal-title":"Journal of Usability Studies"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"volume-title":"How to Measure Survey Reliability and Validity","author":"Litwin Mark S.","key":"e_1_2_1_28_1","unstructured":"Mark S. Litwin . 1995. How to Measure Survey Reliability and Validity . SAGE Publications, UCLA School of Medicine, Los Angeles . Mark S. Litwin. 1995. How to Measure Survey Reliability and Validity. SAGE Publications, UCLA School of Medicine, Los Angeles."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/2336664.2336676"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2452376.2452431"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1805286.1805291"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2047485.2047487"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567663"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 3rd International Workshop on the Web and Databases, in Conjunction with PODS\/SIGMOD","author":"Mihaila George A.","year":"2000","unstructured":"George A. Mihaila , Louiqa Raschid , and Maria-Esther Vidal . 2000 . Using quality of data metadata for source selection and ranking . In Proceedings of the 3rd International Workshop on the Web and Databases, in Conjunction with PODS\/SIGMOD 2000. ACM, New York, 93--98. George A. Mihaila, Louiqa Raschid, and Maria-Esther Vidal. 2000. Using quality of data metadata for source selection and ranking. In Proceedings of the 3rd International Workshop on the Web and Databases, in Conjunction with PODS\/SIGMOD 2000. ACM, New York, 93--98."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2458(01)80027-9"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735471.2735474"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2777823"},{"key":"e_1_2_1_38_1","volume-title":"Fernandes","author":"Osorno-Gutierrez Fernando","year":"2013","unstructured":"Fernando Osorno-Gutierrez , Norman W. Paton , and Alvaro A. A . Fernandes . 2013 . Crowdsourcing feedback for pay--as--you--go data integration. In DBCrowd. CEUR-WS.org, Riva del Garda, Trento, Italy , 32--37. Fernando Osorno-Gutierrez, Norman W. Paton, and Alvaro A. A. Fernandes. 2013. Crowdsourcing feedback for pay--as--you--go data integration. In DBCrowd. CEUR-WS.org, Riva del Garda, Trento, Italy, 32--37."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/505248.506010"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2899403"},{"key":"e_1_2_1_42_1","volume-title":"7th Biennial Conference on Innovative Data Systems Research (CIDR\u201915)","author":"Rekatsinas Theodoros","year":"2015","unstructured":"Theodoros Rekatsinas , Xin Luna Dong , Lise Getoor , and Divesh Srivastava . 2015 . Finding quality in quantity: The challenge of discovering valuable sources for integration . In 7th Biennial Conference on Innovative Data Systems Research (CIDR\u201915) . CIDR, Asilomar, CA, 1--7. Theodoros Rekatsinas, Xin Luna Dong, Lise Getoor, and Divesh Srivastava. 2015. Finding quality in quantity: The challenge of discovering valuable sources for integration. In 7th Biennial Conference on Innovative Data Systems Research (CIDR\u201915). CIDR, Asilomar, CA, 1--7."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610504"},{"key":"e_1_2_1_44_1","volume-title":"Keane","author":"Cort\u00e9s R\u00edos Julio C\u00e9sar","year":"2017","unstructured":"Julio C\u00e9sar Cort\u00e9s R\u00edos , Norman W. Paton , Alvaro A. A. Fernandes , Edward Abel , and John A . Keane . 2017 . Targeted feedback collection applied to multi-criteria source selection. In New Trends in Databases and Information Systems (ADBIS\u201917). Springer , Cham, Nicosia, Cyprus, 136--150. Julio C\u00e9sar Cort\u00e9s R\u00edos, Norman W. Paton, Alvaro A. A. Fernandes, Edward Abel, and John A. Keane. 2017. Targeted feedback collection applied to multi-criteria source selection. In New Trends in Databases and Information Systems (ADBIS\u201917). Springer, Cham, Nicosia, Cyprus, 136--150."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2949689.2949690"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/3019233"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807211"},{"volume-title":"Multimodal Interaction in Image and Video Applications","author":"Villegas Mauricio","key":"e_1_2_1_49_1","unstructured":"Mauricio Villegas , Luis A. Leiva , and Roberto Paredes . 2013. Interactive image retrieval based on relevance feedback . In Multimodal Interaction in Image and Video Applications . Springer-Verlag , Berlin , 83--109. Mauricio Villegas, Luis A. Leiva, and Roberto Paredes. 2013. Interactive image retrieval based on relevance feedback. In Multimodal Interaction in Image and Video Applications. Springer-Verlag, Berlin, 83--109."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0374-x"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113268"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3284934","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3284934","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:57:39Z","timestamp":1750208259000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3284934"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,4]]},"references-count":53,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,3,31]]}},"alternative-id":["10.1145\/3284934"],"URL":"https:\/\/doi.org\/10.1145\/3284934","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"type":"print","value":"1936-1955"},{"type":"electronic","value":"1936-1963"}],"subject":[],"published":{"date-parts":[[2019,1,4]]},"assertion":[{"value":"2017-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}