{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T16:13:29Z","timestamp":1775578409644,"version":"3.50.1"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,4,9]],"date-time":"2019-04-09T00:00:00Z","timestamp":1554768000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1528022"],"award-info":[{"award-number":["CNS-1528022"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Energy's National Nuclear Security Administration","award":["DE-NA-0003525"],"award-info":[{"award-number":["DE-NA-0003525"]}]},{"name":"National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Priv. Secur."],"published-print":{"date-parts":[[2019,5,31]]},"abstract":"<jats:p>Secure email is increasingly being touted as usable by novice users, with a push for adoption based on recent concerns about government surveillance. To determine whether secure email is ready for grassroots adoption, we employ a laboratory user study that recruits pairs of novice users to install and use several of the latest systems to exchange secure messages. We present both quantitative and qualitative results from 28 pairs of novices as they use Private WebMail (Pwm), Tutanota, and Virtru and 10 pairs of novices as they use Mailvelope. Participants report being more at ease with this type of study and better able to cope with mistakes since both participants are \u201con the same page.\u201d We find that users prefer integrated solutions over depot-based solutions and that tutorials are important in helping first-time users. Finally, our results demonstrate that Pretty Good Privacy using manual key management is still unusable for novice users, with 9 of 10 participant pairs failing to complete the study.<\/jats:p>","DOI":"10.1145\/3313761","type":"journal-article","created":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T19:55:16Z","timestamp":1554926116000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["A Usability Study of Four Secure Email Tools Using Paired Participants"],"prefix":"10.1145","volume":"22","author":[{"given":"Scott","family":"Ruoti","sequence":"first","affiliation":[{"name":"University of Tennessee"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeff","family":"Andersen","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luke","family":"Dickinson","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Scott","family":"Heidbrink","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tyler","family":"Monson","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"O'neill","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ken","family":"Reese","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brad","family":"Spendlove","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elham","family":"Vaziripour","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin","family":"Wu","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Zappala","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kent","family":"Seamons","sequence":"additional","affiliation":[{"name":"Brigham Young University, Provo, UT"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,4,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.65"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 1th Symposium on Usable Privacy and Security (SOUPS\u201915)","author":"Atwater Erinn","year":"2015","unstructured":"Erinn Atwater , Cecylia Bocovich , Urs Hengartner , Ed Lank , and Ian Goldberg . 2015 . Leading Johnny to water: Designing for usability and trust . In Proceedings of the 1th Symposium on Usable Privacy and Security (SOUPS\u201915) . 69--88. Erinn Atwater, Cecylia Bocovich, Urs Hengartner, Ed Lank, and Ian Goldberg. 2015. Leading Johnny to water: Designing for usability and trust. In Proceedings of the 1th Symposium on Usable Privacy and Security (SOUPS\u201915). 69--88."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916)","author":"Bai Wei","year":"2016","unstructured":"Wei Bai , Moses Namara , Yichen Qian , Patrick Gage Kelley , Michelle L. Mazurek , and Doowon Kim . 2016 . An inconvenient trust: User attitudes toward security and usability tradeoffs for key-directory encryption systems . In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916) . 113--130. Wei Bai, Moses Namara, Yichen Qian, Patrick Gage Kelley, Michelle L. Mazurek, and Doowon Kim. 2016. An inconvenient trust: User attitudes toward security and usability tradeoffs for key-directory encryption systems. In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916). 113--130."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447310802205776"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2835587.2835589"},{"key":"e_1_2_1_6_1","volume-title":"Usability Evaluation in Industry","author":"Brooke John","unstructured":"John Brooke . 1996. SUS\u2014A quick and dirty usability scale . In Usability Evaluation in Industry . CRC Press , Boca Raton, FL . John Brooke. 1996. SUS\u2014A quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, Boca Raton, FL."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/2817912.2817913"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815675.2815695"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813607"},{"key":"e_1_2_1_10_1","volume-title":"PGP: Pretty Good Privacy. O\u2019Reilly Media","author":"Garfinkel Simson","year":"1995","unstructured":"Simson Garfinkel . 1995 . PGP: Pretty Good Privacy. O\u2019Reilly Media , Inc., Sebastopol, CA. Simson Garfinkel. 1995. PGP: Pretty Good Privacy. O\u2019Reilly Media, Inc., Sebastopol, CA."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSECP.2003.1253564"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1073001.1073003"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124862"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2016.23055"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2017.41"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 24th USENIX Security Symposium (USENIX Security\u201915)","author":"Melara Marcela S.","unstructured":"Marcela S. Melara , Aaron Blankstein , Joseph Bonneau , Michael J. Freedman , and Edward W. Felten . 2015. CONIKS: A privacy-preserving consistent key service for secure end-to-end communication . In Proceedings of the 24th USENIX Security Symposium (USENIX Security\u201915) . 383--398. Marcela S. Melara, Aaron Blankstein, Joseph Bonneau, Michael J. Freedman, and Edward W. Felten. 2015. CONIKS: A privacy-preserving consistent key service for secure end-to-end communication. In Proceedings of the 24th USENIX Security Symposium (USENIX Security\u201915). 383--398."},{"key":"e_1_2_1_17_1","volume-title":"Obedience to Authority","author":"Milgram Stanley","unstructured":"Stanley Milgram and Ernest Van den Haag . 1978. Obedience to Authority . Ziff--Davis Publishing Company , New York, NY . Stanley Milgram and Ernest Van den Haag. 1978. Obedience to Authority. Ziff--Davis Publishing Company, New York, NY."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918)","author":"Poddebniak Damian","year":"2018","unstructured":"Damian Poddebniak , Christian Dresen , Jens M\u00fcller , Fabian Ising , Sebastian Schinzel , Simon Friedberger , Juraj Somorovsky , 2018 . Efail: Breaking S\/MIME and OpenPGP email encryption using exfiltration channels . In Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918) . 549--566. https:\/\/www.usenix.org\/conference\/usenixsecurity18\/presentation\/poddebniak. Damian Poddebniak, Christian Dresen, Jens M\u00fcller, Fabian Ising, Sebastian Schinzel, Simon Friedberger, Juraj Somorovsky, et al. 2018. Efail: Breaking S\/MIME and OpenPGP email encryption using exfiltration channels. In Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918). 549--566. https:\/\/www.usenix.org\/conference\/usenixsecurity18\/presentation\/poddebniak."},{"key":"e_1_2_1_19_1","volume-title":"Retrieved","author":"Ramsdell B.","year":"2019","unstructured":"B. Ramsdell and S. Turner . 2010. Secure\/Multipurpose Internet Mail Extensions (S\/MIME) Version 3.2 Message Specification. RFC 5751 (Proposed Standard) . Retrieved March 19, 2019 from http:\/\/www.ietf.org\/rfc\/rfc5751.txt. B. Ramsdell and S. Turner. 2010. Secure\/Multipurpose Internet Mail Extensions (S\/MIME) Version 3.2 Message Specification. RFC 5751 (Proposed Standard). Retrieved March 19, 2019 from http:\/\/www.ietf.org\/rfc\/rfc5751.txt."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-08506-7_13"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858400"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984580"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Ruoti Scott","year":"2018","unstructured":"Scott Ruoti , Jeff Andersen , Tyler Monson , Daniel Zappala , and Kent Seamons . 2018 . A comparative usability study of key management in secure email . In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918) . 375--394. https:\/\/www.usenix.org\/conference\/soups2018\/presentation\/ruoti. Scott Ruoti, Jeff Andersen, Tyler Monson, Daniel Zappala, and Kent Seamons. 2018. A comparative usability study of key management in secure email. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918). 375--394. https:\/\/www.usenix.org\/conference\/soups2018\/presentation\/ruoti."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501604.2501609"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741683"},{"key":"e_1_2_1_26_1","volume-title":"Benchmarks 8 Best Practices. Measuring Usability LLC","author":"Sauro Jeff","unstructured":"Jeff Sauro . 2011. A Practical Guide to the System Usability Scale: Background , Benchmarks 8 Best Practices. Measuring Usability LLC , Denver, CO . Jeff Sauro. 2011. A Practical Guide to the System Usability Scale: Background, Benchmarks 8 Best Practices. Measuring Usability LLC, Denver, CO."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 14th International Cryptology Conference (Crypto\u201984)","author":"Shamir Adi","year":"1984","unstructured":"Adi Shamir . 1984 . Identity-based cryptosystems and signature schemes . In Proceedings of the 14th International Cryptology Conference (Crypto\u201984) . 47--53. Adi Shamir. 1984. Identity-based cryptosystems and signature schemes. In Proceedings of the 14th International Cryptology Conference (Crypto\u201984). 47--53."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the Poster Session at the Symposium on Usable Privacy and Security.","author":"Sheng S.","unstructured":"S. Sheng , L. Broderick , C. A. Koranda , and J. J. Hyland . 2006. Why Johnny still can\u2019t encrypt: Evaluating the usability of email encryption software . In Proceedings of the Poster Session at the Symposium on Usable Privacy and Security. S. Sheng, L. Broderick, C. A. Koranda, and J. J. Hyland. 2006. Why Johnny still can\u2019t encrypt: Evaluating the usability of email encryption software. In Proceedings of the Poster Session at the Symposium on Usable Privacy and Security."},{"key":"e_1_2_1_29_1","volume-title":"Browser-Based Manual Encryption. Master\u2019s Thesis","author":"Song Yuanzheng","unstructured":"Yuanzheng Song . 2014. Browser-Based Manual Encryption. Master\u2019s Thesis . Brigham Young University, Provo , UT. Yuanzheng Song. 2014. Browser-Based Manual Encryption. Master\u2019s Thesis. Brigham Young University, Provo, UT."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Usable Security Experiment Reports Workshop at the Symposium on Usable Privacy and Security.","author":"Sotirakopoulos Andreas","year":"2010","unstructured":"Andreas Sotirakopoulos , Kirstie Hawkey , and Konstantin Beznosov . 2010 . \u201c I did it because I trusted you\u201d: Challenges with the study environment biasing participant behaviours . In Proceedings of the Usable Security Experiment Reports Workshop at the Symposium on Usable Privacy and Security. Andreas Sotirakopoulos, Kirstie Hawkey, and Konstantin Beznosov. 2010. \u201cI did it because I trusted you\u201d: Challenges with the study environment biasing participant behaviours. In Proceedings of the Usable Security Experiment Reports Workshop at the Symposium on Usable Privacy and Security."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the Usability Professional Association Conference. 1--12","author":"Thomas","unstructured":"Thomas S. Tullis and Jacqueline N. Stetson. 2004. A comparison of questionnaires for assessing website usability . In Proceedings of the Usability Professional Association Conference. 1--12 . Thomas S. Tullis and Jacqueline N. Stetson. 2004. A comparison of questionnaires for assessing website usability. In Proceedings of the Usability Professional Association Conference. 1--12."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 8th USENIX Security Symposium (USENIX Security\u201999)","author":"Whitten A.","unstructured":"A. Whitten and J. D. Tygar . 1999. Why Johnny can\u2019t encrypt: A usability evaluation of PGP 5.0 . In Proceedings of the 8th USENIX Security Symposium (USENIX Security\u201999) . 14--28. A. Whitten and J. D. Tygar. 1999. Why Johnny can\u2019t encrypt: A usability evaluation of PGP 5.0. In Proceedings of the 8th USENIX Security Symposium (USENIX Security\u201999). 14--28."}],"container-title":["ACM Transactions on Privacy and Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3313761","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3313761","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3313761","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:54:33Z","timestamp":1750204473000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3313761"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,9]]},"references-count":32,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,5,31]]}},"alternative-id":["10.1145\/3313761"],"URL":"https:\/\/doi.org\/10.1145\/3313761","relation":{},"ISSN":["2471-2566","2471-2574"],"issn-type":[{"value":"2471-2566","type":"print"},{"value":"2471-2574","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4,9]]},"assertion":[{"value":"2018-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}