{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T11:34:08Z","timestamp":1778758448170,"version":"3.51.4"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T00:00:00Z","timestamp":1634774400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T00:00:00Z","timestamp":1634774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008815","name":"Libera Universit\u00e0 di Bolzano","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008815","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Inf Technol Tourism"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Recommender Systems (RSs) are often assessed in off-line settings by measuring the system precision in predicting the observed user\u2019s ratings or choices. But, when a<jats:italic>precise<\/jats:italic>RS is on-line, the generated recommendations can be perceived as marginally useful because lacking novelty. The underlying problem is that it is hard to build an RS that can correctly generalise, from the analysis of user\u2019s observed behaviour, and can identify the essential characteristics of novel and yet relevant recommendations. In this paper we address the above mentioned issue by considering four RSs that try to excel on different target criteria: precision, relevance and novelty. Two state of the art RSs called and follow a classical Nearest Neighbour approach, while the other two, and are based on Inverse Reinforcement Learning. and optimise precision, tries to identify the characteristics of POIs that make them relevant, and , a novel RS here introduced, is similar to but it also tries to recommend popular POIs. In an off-line experiment we discover that the recommendations produced by and optimise precision essentially by recommending quite popular POIs. can be tuned to achieve a desired level of precision at the cost of losing part of the best capability of to generate novel and yet relevant recommendations. In the on-line study we discover that the recommendations of and are liked more than those produced by . The rationale of that was found in the large percentage of novel recommendations produced by , which are difficult to appreciate. However, excels in recommending items that are both novel and liked by the users.<\/jats:p>","DOI":"10.1007\/s40558-021-00214-5","type":"journal-article","created":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T12:02:36Z","timestamp":1634817756000},"page":"473-508","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Popularity, novelty and relevance in point of interest recommendation: an experimental analysis"],"prefix":"10.1007","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8853-8881","authenticated-orcid":false,"given":"David","family":"Massimo","sequence":"first","affiliation":[]},{"given":"Francesco","family":"Ricci","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,10,21]]},"reference":[{"key":"214_CR1","unstructured":"Abdollahpouri H, Burke R, Mobasher B (2019) Managing popularity bias in recommender systems with personalized re-ranking. In: Bart\u00e1k R, Brawner KW (eds) Proceedings of the thirty-second international Florida artificial intelligence research society conference, Sarasota, Florida, USA, May 19\u201322 2019. AAAI Press, Palo Alto, pp 413\u2013418"},{"issue":"1","key":"214_CR2","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s11257-019-09256-1","volume":"30","author":"H Abdollahpouri","year":"2020","unstructured":"Abdollahpouri H, Adomavicius G, Burke R, Guy I, Jannach D, Kamishima T, Krasnodebski J, Pizzato LA (2020) Multistakeholder recommendation: survey and research directions. User Modeling and User-Adapted Interaction 30(1):127\u2013158","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"214_CR3","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1007\/978-1-4899-7637-6_6","volume-title":"Recommender systems handbook","author":"G Adomavicius","year":"2015","unstructured":"Adomavicius G, Tuzhilin A (2015) Context-aware recommender systems. In: Ricci F, Rokach L, Shapira B (eds) Recommender systems handbook. Springer, Berlin, pp 191\u2013226"},{"key":"214_CR4","unstructured":"Babes M, Marivate VN, Subramanian K, Littman ML (2011) Apprenticeship learning about multiple intentions. In: Getoor L, Scheffer T (eds) Proceedings of the 28th international conference on machine learning, ICML 2011, Bellevue, Washington, USA, June 28\u2013July 2, 2011. Omnipress, Madison, pp 897\u2013904"},{"issue":"4","key":"214_CR5","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1145\/2133806.2133826","volume":"55","author":"DM Blei","year":"2012","unstructured":"Blei DM (2012) Probabilistic topic models. Communications of the ACM 55(4):77\u201384","journal-title":"Communications of the ACM"},{"key":"#cr-split#-214_CR8.1","unstructured":"Cheng C, Yang H, Lyu MR, King I (2013) Where you like to go next: successive point-of-interest recommendation. In: Rossi F"},{"key":"#cr-split#-214_CR8.2","unstructured":"(ed) IJCAI 2013, Proceedings of the 23rd international joint conference on artificial intelligence, Beijing, China, August 3-9, 2013. IJCAI\/AAAI, Palo Alto, pp 2605-2611"},{"key":"214_CR9","doi-asserted-by":"publisher","first-page":"96","DOI":"10.3758\/PBR.15.1.96","volume":"15","author":"A Do","year":"2008","unstructured":"Do A, Rupert A, Wolford G (2008) Evaluations of pleasurable experiences: the peak-end rule. Psychonomic Bulletin & Review 15:96\u201398","journal-title":"Psychonomic Bulletin &amp; Review"},{"key":"214_CR10","doi-asserted-by":"crossref","unstructured":"Ekstrand MD, Harper FM, Willemsen MC, Konstan JA (2014) User perception of differences in recommender algorithms. In: Kobsa A, Zhou MX, Ester M, Koren Y (eds) Eighth ACM conference on recommender systems, RecSys \u201914, Foster City, Silicon Valley, CA, USA, October 06\u201310, 2014. ACM, New York, pp 161\u2013168","DOI":"10.1145\/2645710.2645737"},{"key":"214_CR11","doi-asserted-by":"crossref","unstructured":"Greene D, O\u2019Callaghan D, Cunningham P (2014) How many topics? Stability analysis for topic models. In: Calders T, Esposito F, H\u00fcllermeier E, Meo R (eds) European conference on machine learning and knowledge discovery in databases, ECML PKDD 2014, Nancy, France, September 15\u201319, 2014. Proceedings, Part I. Lecture notes in computer science, vol 8724. Springer, Berlin, pp 498\u2013513","DOI":"10.1007\/978-3-662-44848-9_32"},{"key":"214_CR12","series-title":"Springer series in statistics","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21606-5","volume-title":"The elements of statistical learning","author":"T Hastie","year":"2001","unstructured":"Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning. Springer series in statistics. Springer New York Inc., New York"},{"key":"214_CR13","doi-asserted-by":"crossref","unstructured":"He J, Li X, Liao L, Song D, Cheung WK (2016) Inferring a personalized next point-of-interest recommendation model with latent behavior patterns. In: Dale Schuurmans and Michael\u00a0P. Wellman (eds) Proceedings of the thirtieth AAAI conference on artificial intelligence, February 12\u201317, 2016, Phoenix, Arizona, USA. AAAI Press, Palo Alto, pp 137\u2013143","DOI":"10.1609\/aaai.v30i1.9994"},{"issue":"4","key":"214_CR14","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1504\/IJDMMM.2014.066762","volume":"6","author":"H Huang","year":"2014","unstructured":"Huang H, Gartner G (2014) Using trajectories for collaborative filtering-based POI recommendation. International Journal Data Mining Modeling Management 6(4):333\u2013346","journal-title":"International Journal Data Mining Modeling Management"},{"issue":"5","key":"214_CR15","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1007\/s11257-015-9165-3","volume":"25","author":"D Jannach","year":"2015","unstructured":"Jannach D, Lerche L, Kamehkhosh I, Jugovac M (2015) What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25(5):427\u2013491","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"214_CR16","doi-asserted-by":"crossref","unstructured":"Jannach D, Kamehkhosh I, Lerche L (2017) Leveraging multi-dimensional user models for personalized next-track music recommendation. In: Seffah A, Penzenstadler B, Alves C, Peng X (eds) Proceedings of the symposium on applied computing, SAC 2017, Marrakech, Morocco, April 3\u20137, 2017. ACM, New York, pp 1635\u20131642","DOI":"10.1145\/3019612.3019756"},{"key":"214_CR17","first-page":"510","volume-title":"Recommending based on implicit feedback","author":"D Jannach","year":"2018","unstructured":"Jannach D, Lerche L, Zanker M (2018) Recommending based on implicit feedback. Springer International Publishing, Cham, pp 510\u2013569"},{"key":"214_CR18","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1016\/j.neucom.2016.09.052","volume":"219","author":"X Jia","year":"2017","unstructured":"Jia X, Sun F, Li H, Cao Y, Zhang X (2017) Image multi-label annotation based on supervised nonnegative matrix factorization with new matching measurement. Neurocomputing 219:518\u2013525","journal-title":"Neurocomputing"},{"key":"214_CR19","doi-asserted-by":"crossref","unstructured":"Joachims T (2002) Optimizing search engines using clickthrough data. In: Eighth ACM SIGKDD international conference on knowledge discovery and data mining, SIGKDD 2002, July 23\u201326, 2002, Edmonton, Alberta, Canada. ACM, New York, pp 133\u2013142","DOI":"10.1145\/775047.775067"},{"key":"214_CR20","unstructured":"Knijnenburg BP, Willemsen MC, Broeders R (2014) Smart sustainability through system satisfaction: tailored preference elicitation for energy-saving recommenders. In: 20th Americas conference on information systems, AMCIS 2014, Savannah, Georgia, USA, August 7\u20139, 2014. AIS\/ICIS, pp 1-15"},{"key":"214_CR21","doi-asserted-by":"crossref","unstructured":"Kouki P, Fountalis I, Vasiloglou N, Cui X, Liberty E, Al Jadda K (2020) From the lab to production: a case study of session-based recommendations in the home-improvement domain. In: Santos RLT, Marinho LB, Daly EM, Chen L, Falk K, Koenigstein N, de\u00a0Moura ES (eds) Fourteenth ACM conference on recommender systems, RecSys 2020, Virtual event, Brazil, September 22\u201326, 2020. ACM, New York, pp 140\u2013149","DOI":"10.1145\/3383313.3412235"},{"issue":"6755","key":"214_CR22","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1038\/44565","volume":"401","author":"DD Lee","year":"1999","unstructured":"Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788\u2013791","journal-title":"Nature"},{"key":"214_CR23","doi-asserted-by":"crossref","unstructured":"Loepp B, Donkers T, Kleemann T, Ziegler J (2018) Impact of item consumption on assessment of recommendations in user studies. In: Pera S, Ekstrand MD, Amatriain X, O\u2019Donovan J (eds) 12th ACM conference on recommender systems, RecSys 2018, Vancouver, BC, Canada, October 2\u20137, 2018. ACM, New York, pp 49\u201353","DOI":"10.1145\/3240323.3240375"},{"issue":"4\u20135","key":"214_CR24","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1007\/s11257-018-9209-6","volume":"28","author":"M Ludewig","year":"2018","unstructured":"Ludewig M, Jannach D (2018) Evaluation of session-based recommendation algorithms. User Modeling and User-Adapted Interaction 28(4\u20135):331\u2013390","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"214_CR25","doi-asserted-by":"crossref","unstructured":"MacGlashan J, Babes-Vroman M, desJardins M, Littman ML, Muresan S, Squire S, Tellex S, Arumugam D, Yang L (2015) Grounding English commands to reward functions. In: Kavraki LE, Hsu D, Buchli J (eds) Robotics: science and systems XI, Sapienza University of Rome, Rome, Italy, July 13\u201317, 2015","DOI":"10.15607\/RSS.2015.XI.018"},{"key":"214_CR26","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071","volume-title":"Introduction to information retrieval","author":"CD Manning","year":"2008","unstructured":"Manning CD, Raghavan P, Sch\u00fctze H (2008) Introduction to information retrieval. Cambridge University Press, New York"},{"key":"214_CR7","doi-asserted-by":"crossref","unstructured":"Massimo D, Ricci F (2018) Harnessing a generalised user behaviour model for next-POI recommendation. In: Pera S, Ekstrand MD, Amatrian X, O'Donovan J (eds) 12th ACM conference on recommender systems, RecSys '18, Vancouver, BC, Canada, October 2\u20137, 2018. ACM, New York, pp 402\u2013406","DOI":"10.1145\/3240323.3240392"},{"key":"214_CR6","doi-asserted-by":"crossref","unstructured":"Massimo D, Ricci F (2019) Clustering Users' POIs Visit Trajectories for Next-POI Recommendation. In: Pesonen J, Neidhardt J (eds) Information and Communication Technologies in Tourism 2019, ENTER 2019, Nicosia, Cyprus, January 30-February 1, 2019. Springer, pp 3\u201314","DOI":"10.1007\/978-3-030-05940-8_1"},{"key":"214_CR27","doi-asserted-by":"crossref","unstructured":"Mobasher B, Dai H, Luo T, Nakagawa M (2002) Using sequential and non-sequential patterns in predictive web usage mining tasks. In: Proceedings of the 2002 IEEE international conference on data mining (ICDM 2002), 9\u201312 December 2002, Maebashi City, Japan. IEEE Computer Society, Washington, pp 669\u2013672","DOI":"10.1109\/ICDM.2002.1184025"},{"issue":"1","key":"214_CR28","first-page":"8:1","volume":"7","author":"CI Muntean","year":"2015","unstructured":"Muntean CI, Nardini FM, Silvestri F, Baraglia R (2015) On learning prediction models for tourists paths. ACM Transaction on IntelligentSystems and Technology 7(1):8:1-8:34","journal-title":"ACM Transaction on Intelligent Systems and Technology"},{"key":"#cr-split#-214_CR29.1","unstructured":"Ng AY, Russell SJ (2000) Algorithms for inverse reinforcement learning. In: Langley P"},{"key":"#cr-split#-214_CR29.2","unstructured":"(ed) Proceedings of the seventeenth international conference on machine learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29-July 2, 2000, pp 663-670. Morgan Kaufmann, Burlington"},{"key":"214_CR30","unstructured":"Palumbo E, Rizzo G, Troncy R, Baralis E (2017) Predicting your next stop-over from location-based social network data with recurrent neural networks. In: Neidhardt J, Fesenmaier DR, Kuflik T, W\u00f6rndl W (eds) Proceedings of the 2nd workshop on recommenders in tourism co-located with 11th ACM conference on recommender systems (RecSys 2017), Como, Italy, August 27, 2017. CEUR workshop proceedings, vol 1906, pp 1\u20138"},{"key":"214_CR31","doi-asserted-by":"crossref","unstructured":"Park Y-J, Tuzhilin A (2008) The long tail of recommender systems and how to leverage it. In: Proceedings of the 2008 ACM conference on recommender systems, RecSys \u201908, New York, NY, USA, 23-25 October 2008. Association for Computing Machinery, New York, pp 11\u201318","DOI":"10.1145\/1454008.1454012"},{"key":"214_CR32","doi-asserted-by":"crossref","unstructured":"Perentis C, Vescovi M, Lepri B (2015) Investigating factors affecting personal data disclosure. In: Proceedings of the 24th international conference on world wide web, WWW \u201915 Companion, New York, NY, USA, 2015. ACM, New York, pp 89\u201390","DOI":"10.1145\/2740908.2742757"},{"key":"214_CR33","unstructured":"Poikela M, Schmidt R, Wechsung I, M\u00f6ller S (2014) Locate!-when do users disclose location? In: Workshop on privacy personas and segmentation (PPS) at the tenth symposium on usable privacy and security (SOUPS), Menlo Park, CA, July 9-11, 2014"},{"key":"214_CR34","first-page":"1","volume-title":"Data mining","author":"A Rajaraman","year":"2011","unstructured":"Rajaraman A, Ullman JD (2011) Data mining. Cambridge University Press, Cambridge, pp 1\u201317"},{"key":"214_CR35","unstructured":"Revfine.com (2019) Travel and tourism industry; an complete overview of all activities. Travel and Tourism Industry; An complete Overview of All Activities. www.revfine.com\/travel-and-tourism\/. Accessed 5 May 2019"},{"key":"214_CR36","doi-asserted-by":"crossref","unstructured":"Ricci F, Rokach L, Shapira B (2015) Recommender systems: introduction and challenges. In: Ricci F, Rokach L, Shapira B (eds) Recommender systems handbook. Springer US, Boston, pp 1\u201334","DOI":"10.1007\/978-1-4899-7637-6_1"},{"key":"214_CR37","doi-asserted-by":"publisher","first-page":"102609","DOI":"10.1016\/j.ipm.2021.102609","volume":"58","author":"P Sanchez","year":"2021","unstructured":"Sanchez P, Bellog\u00edn A (2021) On the effects of aggregation strategies for different groups of users in venue recommendation. Information Processing and Management 58:102609","journal-title":"Information Processing and Management"},{"issue":"2","key":"214_CR38","doi-asserted-by":"publisher","first-page":"167","DOI":"10.2307\/249477","volume":"20","author":"HJ Smith","year":"1996","unstructured":"Smith HJ, Milberg SJ, Burke SJ (1996) Information privacy: measuring individuals\u2019 concerns about organizational practices. MIS Quarterly: Management Information Systems 20(2):167\u2013196","journal-title":"MIS Quarterly: Management Information Systems"},{"key":"214_CR39","series-title":"Adaptive computation and machine learning","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.1998.712192","volume-title":"Reinforcement learning\u2014an introduction","author":"RS Sutton","year":"1998","unstructured":"Sutton RS, Barto AG (1998) Reinforcement learning\u2014an introduction. Adaptive computation and machine learning. MIT Press, Cambridge"},{"key":"214_CR40","unstructured":"Suzuki S (2018) Comparative analysis of human movement prediction: space syntax and inverse reinforcement learning. arXiv:1801.00464 [cs.MA]"},{"key":"214_CR41","doi-asserted-by":"crossref","unstructured":"Torrijos S, Bellog\u00edn A, S\u00e1nchez P (2020) Discovering related users in location-based social networks. In: Kuflik T, Torre I, Burke R, Gena C (eds) 28th ACM conference on user modeling, adaptation and personalization, UMAP 2020, Genoa, Italy, July 12\u201318, 2020. ACM, New York, pp 353\u2013357","DOI":"10.1145\/3340631.3394882"},{"key":"#cr-split#-214_CR42.1","doi-asserted-by":"crossref","unstructured":"Wang H, Shen H, Ouyang W, Cheng X (2018) Exploiting POI-specific geographical influence for point-of-interest recommendation. In: Lang J","DOI":"10.24963\/ijcai.2018\/539"},{"key":"#cr-split#-214_CR42.2","unstructured":"(ed) 27th International joint conference on artificial intelligence, IJCAI 2018, Stockholm, Sweden, July 13-19, 2018. IJCAI, pp 3877-3883"},{"key":"214_CR43","volume-title":"Collection of tourism expenditure statistics","author":"World Tourism Organization","year":"1995","unstructured":"World Tourism Organization (1995) Collection of tourism expenditure statistics. World Tourism Organization (UNWTO), Madrid"},{"key":"214_CR44","doi-asserted-by":"crossref","unstructured":"Yang L, Cui Y, Xuan Y, Wang C, Belongie S, Estrin D (2018) Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In: Proceedings of the 12th ACM conference on recommender systems, RecSys \u201918, New York, NY, USA, 2018. Association for Computing Machinery, New York, pp 279\u2013287","DOI":"10.1145\/3240323.3240355"},{"key":"214_CR45","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.ins.2018.04.061","volume":"454\u2013455","author":"C Zhang","year":"2018","unstructured":"Zhang C, Zhang H, Wang J (2018) Personalized restaurant recommendation method combining group correlations and customer preferences. Information Sciences 454\u2013455:128\u2013143","journal-title":"Information Sciences"},{"key":"214_CR46","unstructured":"Ziebart BD, Maas AL, Bagnell JA, Dey AK (2008) Maximum entropy inverse reinforcement learning. In: Fox D, Gomes CP (eds) Proceedings of the twenty-third AAAI conference on artificial intelligence, AAAI 2008, Chicago, Illinois, USA, July 13\u201317, 2008. AAAI Press, Palo Alto, pp 1433\u20131438"}],"container-title":["Information Technology &amp; Tourism"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40558-021-00214-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40558-021-00214-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40558-021-00214-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T04:28:23Z","timestamp":1673584103000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40558-021-00214-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,21]]},"references-count":49,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["214"],"URL":"https:\/\/doi.org\/10.1007\/s40558-021-00214-5","relation":{},"ISSN":["1098-3058","1943-4294"],"issn-type":[{"value":"1098-3058","type":"print"},{"value":"1943-4294","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,21]]},"assertion":[{"value":"17 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 September 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 September 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 October 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"There are no conflicts of interest or competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Users involved the live user study were explicitly informed about the study nature and asked about the consent for participation.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Users involved the live user study were informed about the fact that the collected data were going to be published in an aggregated form.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}