{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T18:44:43Z","timestamp":1755801883593,"version":"3.44.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"IIS","award":["2232551"],"award-info":[{"award-number":["2232551"]}]},{"name":"CNS","award":["2016397"],"award-info":[{"award-number":["2016397"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,11,7]]},"abstract":"<jats:p>Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees' time and reduce their awareness. This study introduces CommTool--a prototype tool to help organizational communicators better understand their emails' performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders' decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.<\/jats:p>","DOI":"10.1145\/3686911","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T15:52:40Z","timestamp":1731081160000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Supporting Organizations in Improving Employee Bulk E-mail --- A Tool Design and Evaluation Study"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0585-0453","authenticated-orcid":false,"given":"Ruoyan","family":"Kong","sequence":"first","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, Minnesota, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4653-0384","authenticated-orcid":false,"given":"Ruixuan","family":"Sun","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, Minnesota, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1027-9733","authenticated-orcid":false,"given":"Charles Chuankai","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, Minnesota, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2988-5001","authenticated-orcid":false,"given":"Ye","family":"Yuan","sequence":"additional","affiliation":[{"name":"McMaster University &amp; University of Minnesota, Hamilton, ON, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7788-2748","authenticated-orcid":false,"given":"Joseph A.","family":"Konstan","sequence":"additional","affiliation":[{"name":"University of Minnesota - Twin Cities, Minneapolis, Minnesota, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/248448.248450"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Gregg Bernstein and Laurissa Wolfram-Hvass. 2015. Research Methodologies Data Collection and Analysis at MailChimp: A Case Study. In Contemporary Research Methods and Data Analytics in the News Industry. IGI Global 170--189.","DOI":"10.4018\/978-1-4666-8580-2.ch010"},{"key":"e_1_2_1_3_1","volume-title":"Contextual design. interactions","author":"Beyer Hugh","year":"1999","unstructured":"Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions, Vol. 6, 1 (1999), 32--42."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206319878254"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Virginia Braun and Victoria Clarke. 2012. Thematic analysis. (2012).","DOI":"10.1037\/13620-004"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/WI.2006.130"},{"key":"e_1_2_1_7_1","volume-title":"CHI'14 Workshop: Personalised Behaviour Change Technologies. ACM Conference on Human Factors in Computing Systems (CHI).","author":"Cecchinato Marta E","year":"2014","unstructured":"Marta E Cecchinato, Jon Bird, and Anna L Cox. 2014. Personalised email tools: a solution to email overload?. In CHI'14 Workshop: Personalised Behaviour Change Technologies. ACM Conference on Human Factors in Computing Systems (CHI)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1057\/palgrave.jt.5740078"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5120\/1645-2213"},{"key":"e_1_2_1_10_1","volume-title":"Evaluating viral marketing: isolating the key criteria. Marketing Intelligence & Planning","author":"Cruz Danilo","year":"2008","unstructured":"Danilo Cruz and Chris Fill. 2008. Evaluating viral marketing: isolating the key criteria. Marketing Intelligence & Planning (2008)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/765891.766073"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1180875.1180941"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1180875.1180941"},{"key":"e_1_2_1_14_1","volume-title":"Haruna Chiroma, Adebayo Olusola Adetunmbi, Opeyemi Emmanuel Ajibuwa, et al.","author":"Dada Emmanuel Gbenga","year":"2019","unstructured":"Emmanuel Gbenga Dada, Joseph Stephen Bassi, Haruna Chiroma, Adebayo Olusola Adetunmbi, Opeyemi Emmanuel Ajibuwa, et al. 2019. Machine learning for email spam filtering: review, approaches and open research problems. Heliyon, Vol. 5, 6 (2019)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/375348.376626"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3188531"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/1553118X.2019.1575830"},{"key":"e_1_2_1_18_1","unstructured":"Google. 2024. File types blocked in Gmail. https:\/\/support.google.com\/mail\/answer\/6590?hl=en#zippy=%2Cmessages-that-dont-have-attachments%2Cmessages-that-have-attachments"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557013"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/0267257X.2010.543018"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJSPM.2008.023685"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30950-2_3"},{"volume-title":"Contextual design: defining customer-centered systems","author":"Holtzblatt Karen","key":"e_1_2_1_23_1","unstructured":"Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defining customer-centered systems. Elsevier."},{"key":"e_1_2_1_24_1","volume-title":"The cost of email interruption. Journal of systems and information technology","author":"Jackson Thomas","year":"2001","unstructured":"Thomas Jackson, Ray Dawson, and Darren Wilson. 2001. The cost of email interruption. Journal of systems and information technology (2001)."},{"volume-title":"Knowledge management and organizational design","author":"Jensen Michael C","key":"e_1_2_1_25_1","unstructured":"Michael C Jensen and William H Meckling. 2009. Specific and general knowledge, and organizational structure. In Knowledge management and organizational design. Routledge, 17--38."},{"key":"e_1_2_1_26_1","volume-title":"Communication: the flow of information. Communication theory","author":"Katz Daniel","year":"2008","unstructured":"Daniel Katz and Robert L Kahn. 2008. Communication: the flow of information. Communication theory (2008), 382--389."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2729.2006.00210.x"},{"volume-title":"The Elgar Companion to Information Economics","author":"Kong Ruoyan","key":"e_1_2_1_28_1","unstructured":"Ruoyan Kong and Joseph A Konstan. 2024. The challenge of organizational bulk email systems: Model and empirical studies. In The Elgar Companion to Information Economics. Edward Elgar Publishing, 407--435."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588015.3588404"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449154"},{"key":"e_1_2_1_31_1","volume-title":"The Challenge of Organizational Bulk Email Systems: Models and Empirical Studies. The Elgar Companion to Information Economics","author":"Konstan Joseph A","year":"2023","unstructured":"Joseph A Konstan and Ruoyan Kong. 2023. The Challenge of Organizational Bulk Email Systems: Models and Empirical Studies. The Elgar Companion to Information Economics. Edward Elgar Publishing (2023)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2166966.2167000"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174160"},{"key":"e_1_2_1_34_1","unstructured":"MailChimp. 2023. Email Marketing Tools. https:\/\/trailhead.salesforce.com\/content\/learn\/modules\/email-send-speed-optimization\/measure-send-speed"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858262"},{"key":"e_1_2_1_36_1","unstructured":"Campaign Monitor. 2012. Campaign monitor. https:\/\/www.campaignmonitor.com\/"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356766711432225"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1080\/1360080X.2015.1019121"},{"key":"e_1_2_1_39_1","volume-title":"Privacy and Trust: First International Conference, HCI-CPT 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26--31, 2019, Proceedings 21","author":"Polst Svenja","year":"2019","unstructured":"Svenja Polst, Patricia Kelbert, and Denis Feth. 2019. Company privacy dashboards: employee needs and requirements. In HCI for Cybersecurity, Privacy and Trust: First International Conference, HCI-CPT 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26--31, 2019, Proceedings 21. Springer, 429--440."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2017.8057231"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v13i5.2100"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1754199"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/985921.985938"},{"key":"e_1_2_1_44_1","unstructured":"Salesforce. 2023. Help and training community -- Email Studio. https:\/\/help.salesforce.com\/s\/articleView?id=sf.mc_es_email_studio.htm&type=5"},{"key":"e_1_2_1_45_1","unstructured":"Stephan Serrano. 2020. 2020 Email Marketing ROI Statistics: Open Rate to Revenue. (2020). https:\/\/www.barilliance.com\/email-marketing-statistics\/"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1057\/bm.2011.23"},{"key":"e_1_2_1_47_1","volume-title":"Organizational communication as an important factor of company success: Case study of Bosnia and Herzegovina. Business intelligence journal","author":"Spaho Kenan","year":"2011","unstructured":"Kenan Spaho. 2011. Organizational communication as an important factor of company success: Case study of Bosnia and Herzegovina. Business intelligence journal, Vol. 4, 2 (2011), 390--393."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/APEIE.2018.8546243"},{"key":"e_1_2_1_49_1","first-page":"87","article-title":"Marketing automation. Bulletin of the Transilvania University of Brasov","volume":"9","author":"Todor Raluca Dania","year":"2016","unstructured":"Raluca Dania Todor. 2016. Marketing automation. Bulletin of the Transilvania University of Brasov. Economic Sciences. Series V, Vol. 9, 2 (2016), 87.","journal-title":"Economic Sciences. Series V"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1080\/10627260903170985"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788602"},{"volume-title":"Ways of Knowing in HCI","author":"Zimmerman John","key":"e_1_2_1_52_1","unstructured":"John Zimmerman and Jodi Forlizzi. 2014. Research through design in HCI. In Ways of Knowing in HCI. Springer, 167--189."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3686911","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3686911","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T00:56:26Z","timestamp":1755737786000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3686911"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,7]]},"references-count":52,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2024,11,7]]}},"alternative-id":["10.1145\/3686911"],"URL":"https:\/\/doi.org\/10.1145\/3686911","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2024,11,7]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}