{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T10:25:14Z","timestamp":1790245514450,"version":"4.1.0"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,7,26]],"date-time":"2019-07-26T00:00:00Z","timestamp":1564099200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2019,8,31]]},"abstract":"<jats:p>\n                    <jats:bold>Context:<\/jats:bold>\n                    End-user service composition (EUSC) is a service-oriented paradigm that aims to empower end users and allow them to compose their own web applications from reusable service components. User studies have been used to evaluate EUSC tools and processes. Such an approach should benefit software development, because incorporating end users\u2019 feedback into software development should make software more useful and usable.\n                    <jats:bold>Problem:<\/jats:bold>\n                    There is a gap in our understanding of what constitutes a user study and how a good user study should be designed, conducted, and reported.\n                    <jats:bold>Goal:<\/jats:bold>\n                    This article aims to address this gap.\n                    <jats:bold>Method:<\/jats:bold>\n                    The article presents a systematic review of 47 selected user studies for EUSC. Guided by a review framework, the article systematically and consistently assesses the focus, methodology and cohesion of each of these studies.\n                    <jats:bold>Results:<\/jats:bold>\n                    The article concludes that the focus of these studies is clear, but their methodology is incomplete and inadequate, their overall cohesion is poor. The findings lead to the development of a design framework and a set of questions for the design, reporting, and review of good user studies for EUSC. The detailed analysis and the insights obtained from the analysis should be applicable to the design of user studies for service-oriented systems as well and indeed for any user studies related to software artifacts.\n                  <\/jats:p>","DOI":"10.1145\/3340294","type":"journal-article","created":{"date-parts":[[2019,7,26]],"date-time":"2019-07-26T09:17:18Z","timestamp":1564132638000},"page":"1-46","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["User Studies on End-User Service Composition"],"prefix":"10.1145","volume":"13","author":[{"given":"Liping","family":"Zhao","sequence":"first","affiliation":[{"name":"School of Computer Science, University of Manchester, Manchester, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pericles","family":"Loucopoulos","sequence":"additional","affiliation":[{"name":"Institute of Digital Innovation and Research, Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evangelia","family":"Kavakli","sequence":"additional","affiliation":[{"name":"Department of Cultural Technology and Communication, University of the Aegean, Mytilene, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keletso J.","family":"Letsholo","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Botswana International University of Science and Technology, Palapye, Botswana"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,7,26]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-55049-2","volume-title":"Mashups: Concepts, Models and Architectures","author":"Daniel F.","year":"2014","unstructured":"F. Daniel and M. Matera . 2014 . Mashups: Concepts, Models and Architectures . Springer . F. Daniel and M. Matera. 2014. Mashups: Concepts, Models and Architectures. Springer."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2014.2304729"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/336512.336523"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/1146351"},{"key":"e_1_2_1_5_1","volume-title":"End-User Development","volume":"9083","author":"Tetteroo D.","unstructured":"D. Tetteroo and P. Markopoulos . 2015. A review of research methods in end user development . In End-User Development , vol. 9083 , P. D\u00edaz, V. Pipek, C. Ardito, C. Jensen, I. Aedo, and A. Boden (Eds.). Springer International Publishing, 58--75. D. Tetteroo and P. Markopoulos. 2015. A review of research methods in end user development. In End-User Development, vol. 9083, P. D\u00edaz, V. Pipek, C. Ardito, C. Jensen, I. Aedo, and A. Boden (Eds.). Springer International Publishing, 58--75."},{"key":"e_1_2_1_6_1","volume-title":"Current Trends in Web Engineering","author":"Namoun A.","unstructured":"A. Namoun , T. Nestler , and A. De Angeli . 2010. Conceptual and usability issues in the composable web of software services . In Current Trends in Web Engineering , vol. 6385 , F. Daniel and F. Facca (Eds.). Springer , Berlin, 396--407. A. Namoun, T. Nestler, and A. De Angeli. 2010. Conceptual and usability issues in the composable web of software services. In Current Trends in Web Engineering, vol. 6385, F. Daniel and F. Facca (Eds.). Springer, Berlin, 396--407."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ECOWS.2010.17"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2008.16"},{"key":"e_1_2_1_9_1","unstructured":"W. M. Newman M. G. Lamming and M. Lamming. 1995. Interactive System Design. Addison-Wesley Reading.  W. M. Newman M. G. Lamming and M. Lamming. 1995. Interactive System Design. Addison-Wesley Reading."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983528"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.159"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1944999.1945009"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2008.110"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the International Symposium on End User Development, 9--24","author":"Cappiello C.","unstructured":"C. Cappiello , F. Daniel , M. Matera , M. Picozzi , and M. Weiss . 2011. Enabling end user development through mashups: requirements, abstractions and innovation toolkits . In Proceedings of the International Symposium on End User Development, 9--24 . C. Cappiello, F. Daniel, M. Matera, M. Picozzi, and M. Weiss. 2011. Enabling end user development through mashups: requirements, abstractions and innovation toolkits. In Proceedings of the International Symposium on End User Development, 9--24."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2008.114"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1497308.1497317"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"F. Daniel and M. Matera. 2014. Mashups and end-user development. In Mashups Springer Berlin 237--268.  F. Daniel and M. Matera. 2014. Mashups and end-user development. In Mashups Springer Berlin 237--268.","DOI":"10.1007\/978-3-642-55049-2_9"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ECOWS.2010.30"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2012.19"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2008.92"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1502650.1502667"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2012.19"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1556460.1556489"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.03.039"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ECOWS.2010.29"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2542168"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993053.1993058"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39200-9_43"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvlc.2013.12.004"},{"key":"e_1_2_1_30_1","unstructured":"ISO\/IEC. 2011. Systems and software engineering\u2014Systems and software quality requirements and evaluation (SQuaRE) \u2014System and software quality models. ISO\/IEC FDIS 25010. Retrieved on from https:\/\/www.iso.org\/standard\/35733.html.  ISO\/IEC. 2011. Systems and software engineering\u2014Systems and software quality requirements and evaluation (SQuaRE) \u2014System and software quality models. ISO\/IEC FDIS 25010. Retrieved on from https:\/\/www.iso.org\/standard\/35733.html."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/2361717"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2519887"},{"key":"e_1_2_1_33_1","unstructured":"J. Rubin and D. Chisnell. 2008. Handbook of Usability Testing: How to Plan Design and Conduct Effective Tests. John Wiley 8 Sons.   J. Rubin and D. Chisnell. 2008. Handbook of Usability Testing: How to Plan Design and Conduct Effective Tests. John Wiley 8 Sons."},{"key":"e_1_2_1_34_1","volume-title":"Usability Engineering","author":"Nielsen J.","unstructured":"J. Nielsen . 1994. Usability Engineering . Elsevier . J. Nielsen. 1994. Usability Engineering. Elsevier."},{"key":"e_1_2_1_35_1","unstructured":"J. S. Dumas and J. Redish. 1999. A Practical Guide to Usability Testing. Intellect Books.   J. S. Dumas and J. Redish. 1999. A Practical Guide to Usability Testing. Intellect Books."},{"key":"e_1_2_1_36_1","volume-title":"Handbook of Human Factors and Ergonomics","author":"Salvendy G.","unstructured":"G. Salvendy . 2012. Handbook of Human Factors and Ergonomics . John Wiley 8 Sons. G. Salvendy. 2012. Handbook of Human Factors and Ergonomics. John Wiley 8 Sons."},{"key":"e_1_2_1_37_1","volume-title":"Handbook of Human Factors and Ergonomics","author":"Lewis J. R.","unstructured":"J. R. Lewis . 2012. Usability testing . In Handbook of Human Factors and Ergonomics , G. Salvendy (Ed.). John Wiley 8 Sons. J. R. Lewis. 2012. Usability testing. In Handbook of Human Factors and Ergonomics, G. Salvendy (Ed.). John Wiley 8 Sons."},{"key":"e_1_2_1_38_1","volume-title":"The Case Study Handbook. Harvard Business School","author":"Gillham B.","unstructured":"B. Gillham . 2007. The Case Study Handbook. Harvard Business School , Boston, MA . B. Gillham. 2007. The Case Study Handbook. Harvard Business School, Boston, MA."},{"key":"e_1_2_1_39_1","first-page":"214","article-title":"Designing a qualitative study","volume":"2","author":"Maxwell J. A.","year":"2008","unstructured":"J. A. Maxwell . 2008 . Designing a qualitative study . SAGE Handbook Appl. Soc. Res. Methods 2 (2008), 214 -- 253 . J. A. Maxwell. 2008. Designing a qualitative study. SAGE Handbook Appl. Soc. Res. Methods 2 (2008), 214--253.","journal-title":"SAGE Handbook Appl. Soc. Res. Methods"},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"S. Easterbrook J. Singer M.-A. Storey and D. Damian. 2008. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering Springer 285--311.  S. Easterbrook J. Singer M.-A. Storey and D. Damian. 2008. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering Springer 285--311.","DOI":"10.1007\/978-1-84800-044-5_11"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9102-8"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-005-1290-x"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1080\/0044929031000"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"J. Kontio J. Bragge and L. Lehtola. 2008. The focus group method as an empirical tool in software engineering. In Guide to Advanced Empirical Software Engineering Springer 93--116.  J. Kontio J. Bragge and L. Lehtola. 2008. The focus group method as an empirical tool in software engineering. In Guide to Advanced Empirical Software Engineering Springer 93--116.","DOI":"10.1007\/978-1-84800-044-5_4"},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"H. Priest P. Roberts and L. Woods. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 1: Theoretical issues. Nurse Res. (Through 2013) 10 (2002) 43.  H. Priest P. Roberts and L. Woods. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 1: Theoretical issues. Nurse Res. (Through 2013) 10 (2002) 43.","DOI":"10.7748\/nr2002.10.10.1.30.c5877"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"L. Woods H. Priest and P. Roberts. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 2: practical illustrations. Nurse Res. (Through 2013) 10 (2002) 43.  L. Woods H. Priest and P. Roberts. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 2: practical illustrations. Nurse Res. (Through 2013) 10 (2002) 43.","DOI":"10.7748\/nr2002.10.10.1.43.c5878"},{"key":"e_1_2_1_48_1","volume-title":"Real World Research","author":"Robson C.","unstructured":"C. Robson and K. McCartan . 2016. Real World Research . Wiley . C. Robson and K. McCartan. 2016. Real World Research. Wiley."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/spip.311"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.391832"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_2_1_52_1","volume-title":"Software Engineering: Experiences from","author":"Wohlin C.","year":"2003","unstructured":"C. Wohlin , M. H\u00f6st , and K. Henningsson . 2003 . Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering: Experiences from ESERNET. Springer , 7--23. C. Wohlin, M. H\u00f6st, and K. Henningsson. 2003. Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering: Experiences from ESERNET. Springer, 7--23."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-007-9053-5"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the NASA SW Engineering Workshop Tutorial, 736--738","author":"Perry D. E.","unstructured":"D. E. Perry , S. E. Sim , and S. M. Easterbrook . 2005. Case studies for software engineers . In Proceedings of the NASA SW Engineering Workshop Tutorial, 736--738 . D. E. Perry, S. E. Sim, and S. M. Easterbrook. 2005. Case studies for software engineers. In Proceedings of the NASA SW Engineering Workshop Tutorial, 736--738."},{"key":"e_1_2_1_55_1","volume-title":"Case Study Research: Design and Methods","author":"Yin R. K.","unstructured":"R. K. Yin . 2013. Case Study Research: Design and Methods . Sage Publications . R. K. Yin. 2013. Case Study Research: Design and Methods. Sage Publications."},{"key":"e_1_2_1_56_1","unstructured":"B. Kitchenham etal 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University Keele Staffs ST5 5BG UK Technical Report EBSE-2007-01 9 July 2007.  B. Kitchenham et al. 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University Keele Staffs ST5 5BG UK Technical Report EBSE-2007-01 9 July 2007."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9492-y"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_2_1_59_1","volume-title":"Case Study Research: Design and Methods","author":"Yin R. K.","unstructured":"R. K. Yin . 1994. Case Study Research: Design and Methods , 2 nd ed. Sage Publications . R. K. Yin. 1994. Case Study Research: Design and Methods, 2nd ed. Sage Publications.","edition":"2"},{"key":"e_1_2_1_60_1","unstructured":"D. M. Fern\u00e1ndez and J.-H. Passoth. 2018. Empirical software engineering: From discipline to interdiscipline. arXiv preprint arXiv:1805.08302.  D. M. Fern\u00e1ndez and J.-H. Passoth. 2018. Empirical software engineering: From discipline to interdiscipline. arXiv preprint arXiv:1805.08302."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9227-7"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering.","author":"Petersen K.","unstructured":"K. Petersen , R. Feldt , S. Mujtaba , and M. Mattsson . 2008. Systematic mapping studies in software engineering . In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745802.2745815"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2785592.2785600"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.04.076"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340294","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340294","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T13:49:32Z","timestamp":1750254572000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340294"}},"subtitle":["A Literature Review and a Design Framework"],"short-title":[],"issued":{"date-parts":[[2019,7,26]]},"references-count":67,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,8,31]]}},"alternative-id":["10.1145\/3340294"],"URL":"https:\/\/doi.org\/10.1145\/3340294","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,26]]},"assertion":[{"value":"2018-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}