{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T16:03:07Z","timestamp":1785427387124,"version":"3.56.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T00:00:00Z","timestamp":1785369600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Digital Society Initiative (DSI) of the University of Zurich"},{"name":"DSI Excellence Program, the Hasler Foundation"},{"name":"Swiss Federal Office of Communications"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2026,12,31]]},"abstract":"<jats:p>This resource article presents Informfully, an open source software project for content distribution to support user studies. It is a free resource for the research community to help conduct online studies investigating the impact of item recommendations on users\u2019 consumption behavior. Informfully allows for the push of text, image, audio, and video content to users and automatically generates a detailed log of their consumption history. It features three core components: (1)\u00a0a front end for displaying and interacting with recommended items, (2)\u00a0a back end for researchers to create and maintain user experiments, and (3)\u00a0a simple JSON-based exchange format for ranked item recommendations to interface with third-party frameworks. We provide a system overview and outline the platform\u2019s three core components. A sample workflow is presented for conducting field studies that incorporate multiple user groups, personalize recommendations, and measure the impact of algorithms on user engagement.<\/jats:p>","DOI":"10.1145\/3820883","type":"journal-article","created":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T20:58:46Z","timestamp":1781211526000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Informfully Research Platform - An Open Source Project for Conducting Empirical Research with Recommender Systems"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7987-8446","authenticated-orcid":false,"given":"Lucien","family":"Heitz","sequence":"first","affiliation":[{"name":"Department of Informatics, University of Zurich","place":["Zurich, Switzerland"]},{"name":"Digital Society Initiative, University of Zurich","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0056-2196","authenticated-orcid":false,"given":"Julian A.","family":"Croci","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Zurich","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9840-7735","authenticated-orcid":false,"given":"Madhav","family":"Sachdeva","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Zurich","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0128-4602","authenticated-orcid":false,"given":"Abraham","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Zurich","place":["Zurich, Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,30]]},"reference":[{"issue":"4","key":"e_1_3_3_2_2","article-title":"Reducing recommender systems biases: An investigation of rating display designs","volume":"43","author":"Adomavicius Gediminas","year":"2019","unstructured":"Gediminas Adomavicius, Jesse Bockstedt, Shawn Curley, and Jingjng Zhang. 2019. Reducing recommender systems biases: An investigation of rating display designs. MIS Quarterly 43, 4 (2019), 19\u201318.","journal-title":"MIS Quarterly"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463245"},{"key":"e_1_3_3_4_2","first-page":"311","volume-title":"Proceedings of the 17th International Conference on Database Systems for Advanced Applications, DASFAA 2012, International Workshops: FlashDB, ITEMS, SNSM, SIM 3, DQDI","author":"Bakaev Maxim","year":"2012","unstructured":"Maxim Bakaev and Tatiana Avdeenko. 2012. User interface design guidelines arrangement in a recommender system with frame ontology. In Proceedings of the 17th International Conference on Database Systems for Advanced Applications, DASFAA 2012, International Workshops: FlashDB, ITEMS, SNSM, SIM 3, DQDI. Springer, 311\u2013322."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3450614.3461682"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-015-0156-0"},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/978-3-319-24592-8_12","volume-title":"Research and Advanced Technology for Digital Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Proceedings 19","author":"Beel Joeran","year":"2015","unstructured":"Joeran Beel and Stefan Langer. 2015. A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In Research and Advanced Technology for Digital Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Proceedings 19. Springer, 153\u2013168."},{"issue":"1","key":"e_1_3_3_8_2","first-page":"43","article-title":"Diversity in news recommendation","volume":"9","author":"Bernstein Abraham","year":"2021","unstructured":"Abraham Bernstein, Claes De Vreese, Natali Helberger, Wolfgang Schulz, Katharina Zweig, Lucien Heitz, Suzanne Tolmeijer, et\u00a0al. 2021. Diversity in news recommendation. Dagstuhl Manifestos 9, 1 (2021), 43\u201361.","journal-title":"Dagstuhl Manifestos"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708319.3727558"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642713"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3610640"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/356810.356816"},{"key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/2516641.2516642","volume-title":"Proceedings of the 2013 International News Recommender Systems Workshop and Challenge","author":"Garcin Florent","year":"2013","unstructured":"Florent Garcin and Boi Faltings. 2013. Pen recsys: A personalized news recommender systems framework. In Proceedings of the 2013 International News Recommender Systems Workshop and Challenge. 3\u20139."},{"key":"e_1_3_3_14_2","volume-title":"Workshop Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization","author":"Guo Guibing","year":"2015","unstructured":"Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. 2015. Librec: A java library for recommender systems. In Workshop Proceedings of the 23rd Conference on User Modeling, Adaptation, and Personalization."},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1145\/3298689.3347014","volume-title":"Proceedings of the 13th ACM Conference on Recommender Systems (RecSys\u201919)","author":"Harambam Jaron","year":"2019","unstructured":"Jaron Harambam, Dimitrios Bountouridis, Mykola Makhortykh, and Joris van Hoboken. 2019. Designing for the better by taking users into account: A qualitative evaluation of user control mechanisms in (news) recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys\u201919). ACM, New York, NY, USA, 69\u201377."},{"key":"e_1_3_3_16_2","volume-title":"Proceedings of the 1st Workshop on the Normative Design and Evaluation of Recommender Systems","author":"Heitz Lucien","year":"2023","unstructured":"Lucien Heitz. 2023. Classification of normative recommender systems. In Proceedings of the 1st Workshop on the Normative Design and Evaluation of Recommender Systems."},{"key":"e_1_3_3_17_2","volume-title":"Working Notes Proceedings of the MediaEval 2023 Workshop","author":"Heitz Lucien","year":"2024","unstructured":"Lucien Heitz, Abraham Bernstein, and Luca Rossetto. 2024. An empirical exploration of perceived similarity between news article texts and images. In Working Notes Proceedings of the MediaEval 2023 Workshop."},{"key":"e_1_3_3_18_2","volume-title":"Working Notes Proceedings of the MediaEval 2023 Workshop","author":"Heitz Lucien","year":"2024","unstructured":"Lucien Heitz, Yuin Kwan Chan, Hongji Li, Kerui Zeng, Abraham Bernstein, and Luca Rossetto. 2024. Prompt-based alignment of headlines and images using openclip. In Working Notes Proceedings of the MediaEval 2023 Workshop."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640457.3688066"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3687151.3687155"},{"key":"e_1_3_3_21_2","first-page":"792","volume-title":"Proceedings of the 19th ACM Conference on Recommender Systems","author":"Heitz Lucien","year":"2025","unstructured":"Lucien Heitz, Runze Li, Oana Inel, and Abraham Bernstein. 2025. Informfully recommenders \u2013 reproducibility framework for diversity-aware intra-session recommendations. In Proceedings of the 19th ACM Conference on Recommender Systems. ACM New York, NY, USA, 792\u2013801."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3608834"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2021.2021804"},{"key":"e_1_3_3_24_2","volume-title":"Proceedings of the 2nd Workshop on the Normative Design and Evaluation of Recommender Systems","author":"Heitz Lucien","year":"2024","unstructured":"Lucien Heitz, Nicolas Mattis, Oana Inel, and Wouter van Atteveldt. 2024. IDEA \u2013 informfully dataset with enhanced attributes. In Proceedings of the 2nd Workshop on the Normative Design and Evaluation of Recommender Systems."},{"key":"e_1_3_3_25_2","volume-title":"Working Notes Proceedings of the MediaEval 2025 Workshop","author":"Heitz Lucien","year":"2025","unstructured":"Lucien Heitz, Luca Rossetto, Benjamin Kille, Andreas Lommatzsch, Mehdi Elahi, and Duc-Tien Dang-Nguyen. 2025. NewsImages in mediaeval 2025 \u2013 comparing image retrieval and generation for news articles. In Working Notes Proceedings of the MediaEval 2025 Workshop."},{"key":"e_1_3_3_26_2","first-page":"56","volume-title":"AI in Content Curation and Media Pluralism","author":"Heitz Lucien","year":"2021","unstructured":"Lucien Heitz, Krisztina Rozgonyi, Bojana Kostic, Deniz Wagner, and Julia Haas. 2021. AI in Content Curation and Media Pluralism. Organization for Security and Co-operation in Europe, Austria, Chapter 2, 56\u201370. Retrieved from https:\/\/zora.uzh.ch\/id\/eprint\/213723"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2019.1623700"},{"key":"e_1_3_3_28_2","volume-title":"Proceedings of Beyond Algorithms: Reclaiming the Interdisciplinary Roots of Recommender Systems Workshop","author":"Higley Karl","year":"2025","unstructured":"Karl Higley, Robin Burke, Michael D. Ekstrand, and Bart P. Knijnenburg. 2025. What news recommendation research did (but mostly didn\u2019t) teach us about building a news recommender. In Proceedings of Beyond Algorithms: Reclaiming the Interdisciplinary Roots of Recommender Systems Workshop."},{"issue":"6","key":"e_1_3_3_29_2","first-page":"72","article-title":"Mobile application and its global impact","volume":"10","author":"Islam Rashedul","year":"2010","unstructured":"Rashedul Islam, Rofiqul Islam, and Tohidul Mazumder. 2010. Mobile application and its global impact. International Journal of Engineering & Technology 10, 6 (2010), 72\u201378.","journal-title":"International Journal of Engineering & Technology"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v41i4.5312"},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1007\/978-3-642-32273-0_7","volume-title":"E-Commerce and Web Technologies: 13th International Conference, EC-Web 2012. Proceedings 13","author":"Jannach Dietmar","year":"2012","unstructured":"Dietmar Jannach, Markus Zanker, Mouzhi Ge, and Marian Gr\u00f6ning. 2012. Recommender systems in computer science and information systems\u2013a landscape of research. In E-Commerce and Web Technologies: 13th International Conference, EC-Web 2012. Proceedings 13. Springer, 76\u201387."},{"key":"e_1_3_3_32_2","unstructured":"R. Leuener. 2017. NZZ Companion: How we successfully developed a personalised news application. Medium."},{"key":"e_1_3_3_33_2","first-page":"558","volume-title":"Proceedings of the 19th ACM Conference on Recommender Systems","author":"Li Runze","year":"2025","unstructured":"Runze Li, Lucien Heitz, Oana Inel, and Abraham Bernstein. 2025. D-RDW: Diversity-driven random walks for news recommender systems. In Proceedings of the 19th ACM Conference on Recommender Systems. ACM New York, NY, USA, 558\u2013563."},{"issue":"1","key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.5117\/CCR2020.1.003.LOEC","article-title":"3bij3\u2013Developing a framework for researching recommender systems and their effects","volume":"2","author":"Loecherbach Felicia","year":"2020","unstructured":"Felicia Loecherbach and Damian Trilling. 2020. 3bij3\u2013Developing a framework for researching recommender systems and their effects. Computational Communication Research 2, 1 (2020), 53\u201379.","journal-title":"Computational Communication Research"},{"issue":"5","key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1093\/joc\/jqaf019","article-title":"Nudges for news recommenders: Prominent article positioning increases selection, engagement, and recall of environmental news, but reducing complexity does not","volume":"75","author":"Mattis Nicolas","year":"2025","unstructured":"Nicolas Mattis, Lucien Heitz, Philipp K. Masur, Judith Moeller, and Wouter van Atteveldt. 2025. Nudges for news recommenders: Prominent article positioning increases selection, engagement, and recall of environmental news, but reducing complexity does not. Journal of Communication 75, 5 (2025), 437\u2013449.","journal-title":"Journal of Communication"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340631.3394844"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445911"},{"issue":"4","key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/s00530-010-0190-0","article-title":"The effects of recommendations\u2019 presentation on persuasion and satisfaction in a movie recommender system","volume":"16","author":"Nanou Theodora","year":"2010","unstructured":"Theodora Nanou, George Lekakos, and Konstantinos Fouskas. 2010. The effects of recommendations\u2019 presentation on persuasion and satisfaction in a movie recommender system. Multimedia Systems 16, 4 (2010), 219\u2013230.","journal-title":"Multimedia Systems"},{"key":"e_1_3_3_39_2","volume-title":"Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps","author":"Neil Theresa","year":"2014","unstructured":"Theresa Neil. 2014. Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps. \u201d O\u2019Reilly Media, Inc.\u201d."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-77471-6_10#citeas"},{"issue":"1","key":"e_1_3_3_41_2","first-page":"6787504","article-title":"Usability studies on mobile user interface design patterns: A systematic literature review","volume":"2017","author":"Punchoojit Lumpapun","year":"2017","unstructured":"Lumpapun Punchoojit and Nuttanont Hongwarittorrn. 2017. Usability studies on mobile user interface design patterns: A systematic literature review. Advances in Human-Computer Interaction 2017, 1 (2017), 6787504.","journal-title":"Advances in Human-Computer Interaction"},{"key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1007\/978-3-030-67835-7_38","volume-title":"Proceedings of the International Conference on Multimedia Modeling","author":"Rossetto Luca","year":"2021","unstructured":"Luca Rossetto, Matthias Baumgartner, Narges Ashena, Florian Ruosch, Romana Pernisch, Lucien Heitz, and Abraham Bernstein. 2021. VideoGraph\u2013towards using knowledge graphs for interactive video retrieval. In Proceedings of the International Conference on Multimedia Modeling. Springer, 417\u2013422."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3463948.3469068"},{"issue":"95","key":"e_1_3_3_44_2","first-page":"1","article-title":"Cornac: A comparative framework for multimodal recommender systems","volume":"21","author":"Salah Aghiles","year":"2020","unstructured":"Aghiles Salah, Quoc-Tuan Truong, and Hady W. Lauw. 2020. Cornac: A comparative framework for multimodal recommender systems. Journal of Machine Learning Research 21, 95 (2020), 1\u20135.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_45_2","volume-title":"Spotlight on Artificial Intelligence and Freedom of Expression: A Policy Manual","author":"Sargeant Holli","year":"2022","unstructured":"Holli Sargeant, Eliska Pirkova, Matthias C. Kettemann, Marlena Wisniak, Martin Scheinin, Emmi Bevensee, Katie Pentney, Lorna Woods, Lucien Heitz, Krisztina Rozgonyi, et\u00a0al. 2022. Spotlight on Artificial Intelligence and Freedom of Expression: A Policy Manual. Organization for Security and Co-operation in Europe, Austria. Retrieved from https:\/\/osce.org\/representative-on-freedom-of-media\/510332"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196727"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/503271.503224"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3412489"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/1639714.1639777"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113452"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3820883","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T15:38:58Z","timestamp":1785425938000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3820883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,30]]},"references-count":49,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,12,31]]}},"alternative-id":["10.1145\/3820883"],"URL":"https:\/\/doi.org\/10.1145\/3820883","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,30]]},"assertion":[{"value":"2025-03-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}