{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T06:30:56Z","timestamp":1760596256272,"version":"3.41.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2016,6,25]],"date-time":"2016-06-25T00:00:00Z","timestamp":1466812800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2016,8,22]]},"abstract":"<jats:p>Pattern mining, that is, the automated discovery of patterns from data, is a mathematically complex and computationally demanding problem that is generally not manageable by humans. In this article, we focus on small datasets and study whether it is possible to mine patterns with the help of the crowd by means of a set of controlled experiments on a common crowdsourcing platform. We specifically concentrate on mining model patterns from a dataset of real mashup models taken from Yahoo! Pipes and cover the entire pattern mining process, including pattern identification and quality assessment. The results of our experiments show that a sensible design of crowdsourcing tasks indeed may enable the crowd to identify patterns from small datasets (40 models). The results, however, also show that the design of tasks for the assessment of the quality of patterns to decide which patterns to retain for further processing and use is much harder (our experiments fail to elicit assessments from the crowd that are similar to those by an expert). The problem is relevant in general to model-driven development (e.g., UML, business processes, scientific workflows), in that reusable model patterns encode valuable modeling and domain knowledge, such as best practices, organizational conventions, or technical choices, that modelers can benefit from when designing their own models.<\/jats:p>","DOI":"10.1145\/2903138","type":"journal-article","created":{"date-parts":[[2016,6,27]],"date-time":"2016-06-27T15:39:29Z","timestamp":1467041969000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Mining and Quality Assessment of Mashup Model Patterns with the Crowd"],"prefix":"10.1145","volume":"16","author":[{"given":"Carlos","family":"Rodr\u00edguez","sequence":"first","affiliation":[{"name":"University of Trento, Povo (TN), Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Florian","family":"Daniel","sequence":"additional","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fabio","family":"Casati","sequence":"additional","affiliation":[{"name":"University of Trento, Povo (TN), Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,6,25]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Web Services: Concepts, Architectures, and Applications","author":"Alonso Gustavo","year":"2003","unstructured":"Gustavo Alonso , Fabio Casati , Harumi Kuno , and Vijay Machiraju . 2003 . Web Services: Concepts, Architectures, and Applications . Springer . Gustavo Alonso, Fabio Casati, Harumi Kuno, and Vijay Machiraju. 2003. Web Services: Concepts, Architectures, and Applications. Springer."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/2027776.2027788"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2010.60"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/QUATIC.2012.50"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-89652-4_25"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687602"},{"key":"e_1_2_2_7_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-55049-2","volume-title":"Mashups: Concepts, Models and Architectures","author":"Daniel Florian","year":"2014","unstructured":"Florian Daniel and Maristella Matera . 2014 . Mashups: Concepts, Models and Architectures . Springer . Florian Daniel and Maristella Matera. 2014. Mashups: Concepts, Models and Architectures. Springer."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2008.06.012"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2010.09.006"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2008.128"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1132960.1132963"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866718"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687689"},{"key":"e_1_2_2_15_1","volume-title":"Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business","author":"Howe Jeff","year":"2008","unstructured":"Jeff Howe . 2008 . Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business . Crown Publishing Group , New York, NY . Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2009.46"},{"key":"e_1_2_2_17_1","volume-title":"The AAAI Conference on Human Computation (AAAIWS\u201911)","author":"Jung Hyun Joon","year":"2011","unstructured":"Hyun Joon Jung and Matthew Lease . 2011 . Improving consensus accuracy via Z-score and weighted voting . In The AAAI Conference on Human Computation (AAAIWS\u201911) . 88--90. Hyun Joon Jung and Matthew Lease. 2011. Improving consensus accuracy via Z-score and weighted voting. In The AAAI Conference on Human Computation (AAAIWS\u201911). 88--90."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCS.2012.6213338"},{"key":"e_1_2_2_19_1","volume-title":"The Neural Information Processing Systems Workshop on Computational Social Science and the Wisdom of Crowds (NISP\u201911)","author":"Khattak Faiza Khan","year":"2011","unstructured":"Faiza Khan Khattak and Ansaf Salleb-Aouissi . 2011 . Quality control of crowd labeling through expert evaluation . In The Neural Information Processing Systems Workshop on Computational Social Science and the Wisdom of Crowds (NISP\u201911) . Faiza Khan Khattak and Ansaf Salleb-Aouissi. 2011. Quality control of crowd labeling through expert evaluation. In The Neural Information Processing Systems Workshop on Computational Social Science and the Wisdom of Crowds (NISP\u201911)."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2578903.2601079"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2228360.2228590"},{"volume-title":"The Conference on Innovative Data Systems Research (CIDR\u201911)","author":"Marcus Adam","key":"e_1_2_2_23_1","unstructured":"Adam Marcus , Eugene Wu , David R. Karger , Samuel Madden , and Robert C. Miller . 2011. Crowdsourced databases: Query processing with people . In The Conference on Innovative Data Systems Research (CIDR\u201911) . 211--214. Adam Marcus, Eugene Wu, David R. Karger, Samuel Madden, and Robert C. Miller. 2011. Crowdsourced databases: Query processing with people. In The Conference on Innovative Data Systems Research (CIDR\u201911). 211--214."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809400.1809422"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.233837"},{"key":"e_1_2_2_26_1","volume-title":"The SIGIR Workshop on Crowdsourcing for Search Evaluation (CSE\u201910)","author":"McCreadie Richard M. C.","year":"2010","unstructured":"Richard M. C. McCreadie , Craig Macdonald , and Iadh Ounis . 2010 . Crowdsourcing a news query classification dataset . In The SIGIR Workshop on Crowdsourcing for Search Evaluation (CSE\u201910) . 31--38. Richard M. C. McCreadie, Craig Macdonald, and Iadh Ounis. 2010. Crowdsourcing a news query classification dataset. In The SIGIR Workshop on Crowdsourcing for Search Evaluation (CSE\u201910). 31--38."},{"key":"e_1_2_2_27_1","unstructured":"OMG. 2011. Business Process Model and Notation (BPMN) version 2.0. http:\/\/www.bpmn.org. (2011).  OMG. 2011. Business Process Model and Notation (BPMN) version 2.0. http:\/\/www.bpmn.org. (2011)."},{"key":"e_1_2_2_28_1","first-page":"1","article-title":"The Interaction Flow Modeling Language (IFML)","volume":"974","author":"Object Management Group (OMG).","year":"2014","unstructured":"Object Management Group (OMG). 2014 . The Interaction Flow Modeling Language (IFML) , Version 974 1 .0. OMG standard specification. Object Management Group, http:\/\/www.ifml.org. Object Management Group (OMG). 2014. The Interaction Flow Modeling Language (IFML), Version 974 1.0. OMG standard specification. Object Management Group, http:\/\/www.ifml.org.","journal-title":"Version"},{"key":"e_1_2_2_29_1","unstructured":"OMG. 2014. Unified Modeling Language (UML). http:\/\/www.uml.org\/. (2014).  OMG. 2014. Unified Modeling Language (UML). http:\/\/www.uml.org\/. (2014)."},{"key":"e_1_2_2_30_1","volume-title":"The Conference on Innovative Data Systems Research (CIDR\u201911)","author":"Parameswaran Aditya","year":"2011","unstructured":"Aditya Parameswaran and Neoklis Polyzotis . 2011 . Answering queries using humans, algorithms and databases . In The Conference on Innovative Data Systems Research (CIDR\u201911) . 160--166. Aditya Parameswaran and Neoklis Polyzotis. 2011. Answering queries using humans, algorithms and databases. In The Conference on Innovative Data Systems Research (CIDR\u201911). 160--166."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367602"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10172-9_4"},{"key":"e_1_2_2_33_1","volume-title":"Florian Daniel, Hamid R. Motahari Nezhad, and Fabio Casati.","author":"Rodr\u00edguez Carlos","year":"2014","unstructured":"Carlos Rodr\u00edguez , Soudip Roy Chowdhury , Florian Daniel, Hamid R. Motahari Nezhad, and Fabio Casati. 2014 b. Assisted mashup development: On the discovery and recommendation of mashup composition knowledge. In Web Services Foundations. Springer , 683--708. Carlos Rodr\u00edguez, Soudip Roy Chowdhury, Florian Daniel, Hamid R. Motahari Nezhad, and Fabio Casati. 2014b. Assisted mashup development: On the discovery and recommendation of mashup composition knowledge. In Web Services Foundations. Springer, 683--708."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663500"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852786.1852832"},{"volume-title":"Introduction to Data Mining","author":"Tan Pang-Ning","key":"e_1_2_2_37_1","unstructured":"Pang-Ning Tan , Michael Steinbach , and Vipin Kumar . 2005. Introduction to Data Mining . Addison-Wesley . Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2005. Introduction to Data Mining. Addison-Wesley."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2746353"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"volume-title":"Languages, Architectures","author":"Weske Mathias","key":"e_1_2_2_41_1","unstructured":"Mathias Weske . 2007. Business Process Management: Concepts , Languages, Architectures . Springer . Mathias Weske. 2007. Business Process Management: Concepts, Languages, Architectures. Springer."}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2903138","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2903138","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:56:15Z","timestamp":1750222575000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2903138"}},"subtitle":["A Feasibility Study"],"short-title":[],"issued":{"date-parts":[[2016,6,25]]},"references-count":41,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2016,8,22]]}},"alternative-id":["10.1145\/2903138"],"URL":"https:\/\/doi.org\/10.1145\/2903138","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"type":"print","value":"1533-5399"},{"type":"electronic","value":"1557-6051"}],"subject":[],"published":{"date-parts":[[2016,6,25]]},"assertion":[{"value":"2016-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-06-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}