{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T10:54:25Z","timestamp":1776855265898,"version":"3.51.2"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,12,4]],"date-time":"2018-12-04T00:00:00Z","timestamp":1543881600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1330596"],"award-info":[{"award-number":["CNS-1330596"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2019,2,28]]},"abstract":"<jats:p>Website privacy policies are often long and difficult to understand. While research shows that Internet users care about their privacy, they do not have the time to understand the policies of every website they visit, and most users hardly ever read privacy policies. Some recent efforts have aimed to use a combination of crowdsourcing, machine learning, and natural language processing to interpret privacy policies at scale, thus producing annotations for use in interfaces that inform Internet users of salient policy details. However, little attention has been devoted to studying the accuracy of crowdsourced privacy policy annotations, how crowdworker productivity can be enhanced for such a task, and the levels of granularity that are feasible for automatic analysis of privacy policies. In this article, we present a trajectory of work addressing each of these topics. We include analyses of crowdworker performance, evaluation of a method to make a privacy-policy oriented task easier for crowdworkers, a coarse-grained approach to labeling segments of policy text with descriptive themes, and a fine-grained approach to identifying user choices described in policy text. Together, the results from these efforts show the effectiveness of using automated and semi-automated methods for extracting from privacy policies the data practice details that are salient to Internet users\u2019 interests.<\/jats:p>","DOI":"10.1145\/3230665","type":"journal-article","created":{"date-parts":[[2018,12,4]],"date-time":"2018-12-04T15:32:40Z","timestamp":1543937560000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":36,"title":["Analyzing Privacy Policies at Scale"],"prefix":"10.1145","volume":"13","author":[{"given":"Shomir","family":"Wilson","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Florian","family":"Schaub","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frederick","family":"Liu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kanthashree Mysore","family":"Sathyendra","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Smullen","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian","family":"Zimmeck","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rohan","family":"Ramanath","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Story","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fei","family":"Liu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norman","family":"Sadeh","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noah A.","family":"Smith","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,12,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/336992.336995"},{"key":"e_1_2_1_2_1","volume-title":"Smith","author":"Ammar Waleed","year":"2012","unstructured":"Waleed Ammar , Shomir Wilson , Norman Sadeh , and Noah A . Smith . 2012 . Automatic categorization of privacy policies: A pilot study. Technical Report. Carnegie Mellon University . Waleed Ammar, Shomir Wilson, Norman Sadeh, and Noah A. Smith. 2012. Automatic categorization of privacy policies: A pilot study. Technical Report. Carnegie Mellon University."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531653"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2425327.2425330"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 2016 IEEE 24th International Requirements Engineering Conference (RE). 26--35","author":"Bhatia Jaspreet","unstructured":"Jaspreet Bhatia , Travis D. Breaux , Joel R. Reidenberg , and Thomas B. Norton . 2016b. A theory of vagueness and privacy risk perception . In Proceedings of the 2016 IEEE 24th International Requirements Engineering Conference (RE). 26--35 . Jaspreet Bhatia, Travis D. Breaux, Joel R. Reidenberg, and Thomas B. Norton. 2016b. A theory of vagueness and privacy risk perception. In Proceedings of the 2016 IEEE 24th International Requirements Engineering Conference (RE). 26--35."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2907942"},{"key":"e_1_2_1_7_1","first-page":"993","article-title":"Latent dirichlet allocation","author":"Blei David M.","year":"2003","unstructured":"David M. Blei , Andrew Y. Ng , and Michael I. Jordan . 2003 . Latent dirichlet allocation . Journal of Machine Learning Research 3 , Jan (2003), 993 -- 1022 . David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3, Jan (2003), 993--1022.","journal-title":"Journal of Machine Learning Research 3"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 22nd IEEE International Requirements Engineering Conference (RE\u201914)","author":"Travis","unstructured":"Travis D. Breaux and Florian Schaub. 2014. Scaling requirements extraction to the crowd . In Proceedings of the 22nd IEEE International Requirements Engineering Conference (RE\u201914) . IEEE Society Press, Washington, D.C. Travis D. Breaux and Florian Schaub. 2014. Scaling requirements extraction to the crowd. In Proceedings of the 22nd IEEE International Requirements Engineering Conference (RE\u201914). IEEE Society Press, Washington, D.C."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2010.84"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_11_1","volume-title":"KDD Workshop on Data Mining for Social Good.","author":"Chundi Parvathi","unstructured":"Parvathi Chundi and Pranav M. Subramaniam . 2014. An approach to analyze web privacy policy documents . In KDD Workshop on Data Mining for Social Good. Parvathi Chundi and Pranav M. Subramaniam. 2014. An approach to analyze web privacy policy documents. In KDD Workshop on Data Mining for Social Good."},{"key":"e_1_2_1_12_1","volume-title":"Data Privacy Management and Autonomous Spontaneous Security (Lecture Notes in Computer Science)","author":"Costante Elisa","unstructured":"Elisa Costante , Jerry den Hartog , and Milan Petkovi\u0107 . 2013. What websites know about you: Privacy policy analysis using information extraction . In Data Privacy Management and Autonomous Spontaneous Security (Lecture Notes in Computer Science) , Roberto Di Pietro, Javier Herranz, Ernesto Damiani, and Radu State (Eds.), Vol. 7731 . Springer , 146--159. Elisa Costante, Jerry den Hartog, and Milan Petkovi\u0107. 2013. What websites know about you: Privacy policy analysis using information extraction. In Data Privacy Management and Autonomous Spontaneous Security (Lecture Notes in Computer Science), Roberto Di Pietro, Javier Herranz, Ernesto Damiani, and Radu State (Eds.), Vol. 7731. Springer, 146--159."},{"key":"e_1_2_1_13_1","volume-title":"The Platform for Privacy Preferences 1.1 (P3P1.1) Specification","author":"Cranor Lorrie","year":"2018","unstructured":"Lorrie Cranor , B. Dobbs , S. Egelman , G. Hogben , J. Humphrey , M. Langheinrich , M. Marchiori , M. Presler-Marshall , J. Reagle , D. A. Stampley , Matthias Schunter , and Rigo Wenning . 2006. The Platform for Privacy Preferences 1.1 (P3P1.1) Specification . Working Group Note . W3C. Retrieved March 12, 2018 from http:\/\/www.w3.org\/TR\/P3P11\/. Lorrie Cranor, B. Dobbs, S. Egelman, G. Hogben, J. Humphrey, M. Langheinrich, M. Marchiori, M. Presler-Marshall, J. Reagle, D. A. Stampley, Matthias Schunter, and Rigo Wenning. 2006. The Platform for Privacy Preferences 1.1 (P3P1.1) Specification. Working Group Note. W3C. Retrieved March 12, 2018 from http:\/\/www.w3.org\/TR\/P3P11\/."},{"key":"e_1_2_1_14_1","volume-title":"Necessary but not sufficient: Standardized mechanisms for privacy notice and choice. J. on Telecomm. 8 High Tech. L. 10","author":"Cranor Lorrie Faith","year":"2012","unstructured":"Lorrie Faith Cranor . 2012. Necessary but not sufficient: Standardized mechanisms for privacy notice and choice. J. on Telecomm. 8 High Tech. L. 10 ( 2012 ), 273. Lorrie Faith Cranor. 2012. Necessary but not sufficient: Standardized mechanisms for privacy notice and choice. J. on Telecomm. 8 High Tech. L. 10 (2012), 273."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911988"},{"key":"e_1_2_1_16_1","volume-title":"Personalized privacy assistants for the Internet of Things","author":"Das Anupam","year":"2018","unstructured":"Anupam Das , Martin Degeling , Daniel Smullen , and Norman Sadeh . 2018. Personalized privacy assistants for the Internet of Things . IEEE Pervasive Computing\u2014Special Issue on Securing the IoT. 17, 3 ( 2018 ), 35--46. Anupam Das, Martin Degeling, Daniel Smullen, and Norman Sadeh. 2018. Personalized privacy assistants for the Internet of Things. IEEE Pervasive Computing\u2014Special Issue on Securing the IoT. 17, 3 (2018), 35--46."},{"key":"e_1_2_1_17_1","unstructured":"Nick Doty Heather West Justin Brookman Sean Harvey and Erica Newland. 2016. Tracking compliance and scope. Candidate Recommendation. W3C.  Nick Doty Heather West Justin Brookman Sean Harvey and Erica Newland. 2016. Tracking compliance and scope. Candidate Recommendation. W3C."},{"key":"e_1_2_1_18_1","volume-title":"Internationale Tagung Wirtschaftsinformatik (Wirtschaftsinformatik","author":"Ermakova Tatiana","year":"2015","unstructured":"Tatiana Ermakova , Benjamin Fabian , and Eleonora Babina . 2015. Readability of privacy policies of healthcare websites. In 12 . Internationale Tagung Wirtschaftsinformatik (Wirtschaftsinformatik 2015 ). Tatiana Ermakova, Benjamin Fabian, and Eleonora Babina. 2015. Readability of privacy policies of healthcare websites. In 12. Internationale Tagung Wirtschaftsinformatik (Wirtschaftsinformatik 2015)."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE). 312--321","author":"Evans Morgan C.","unstructured":"Morgan C. Evans , Jaspreet Bhatia , Sudarshan Wadkar , and Travis D. Breaux . 2017. An evaluation of constituency-based hyponymy extraction from privacy policies . In Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE). 312--321 . Morgan C. Evans, Jaspreet Bhatia, Sudarshan Wadkar, and Travis D. Breaux. 2017. An evaluation of constituency-based hyponymy extraction from privacy policies. In Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE). 312--321."},{"key":"e_1_2_1_20_1","volume-title":"Privacy Online: A Report to Congress. Technical Report. Federal Trade Commission.","author":"Federal Trade Commission","year":"2000","unstructured":"Federal Trade Commission . 2000 . Privacy Online: A Report to Congress. Technical Report. Federal Trade Commission. Federal Trade Commission. 2000. Privacy Online: A Report to Congress. Technical Report. Federal Trade Commission."},{"key":"e_1_2_1_21_1","volume-title":"Retrieved","author":"Federal Trade Commission","year":"2012","unstructured":"Federal Trade Commission . 2012 . Protecting Consumer Privacy in an Era of Rapid Change: Recommendations For Businesses and Policymakers . Retrieved March 12, 2018 from https:\/\/www.ftc.gov\/reports\/protecting-consumer-privacy-era-rapid-change-recommendations-businesses-policymakers. Federal Trade Commission. 2012. Protecting Consumer Privacy in an Era of Rapid Change: Recommendations For Businesses and Policymakers. Retrieved March 12, 2018 from https:\/\/www.ftc.gov\/reports\/protecting-consumer-privacy-era-rapid-change-recommendations-businesses-policymakers."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2388632.2388647"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 2016 AAAI Fall Symposium Series.","author":"Hosseini Mitra Bokaei","year":"2016","unstructured":"Mitra Bokaei Hosseini , Sudarshan Wadkar , Travis D. Breaux , and Jianwei Niu . 2016 . Lexical similarity of information type hypernyms, meronyms and synonyms in privacy policies . In Proceedings of the 2016 AAAI Fall Symposium Series. Mitra Bokaei Hosseini, Sudarshan Wadkar, Travis D. Breaux, and Jianwei Niu. 2016. Lexical similarity of information type hypernyms, meronyms and synonyms in privacy policies. In Proceedings of the 2016 AAAI Fall Symposium Series."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985752"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370020903586662"},{"key":"e_1_2_1_26_1","volume-title":"Convolutional neural networks for sentence classification. arXiv Preprint arXiv:1408.5882","author":"Kim Yoon","year":"2014","unstructured":"Yoon Kim . 2014. Convolutional neural networks for sentence classification. arXiv Preprint arXiv:1408.5882 ( 2014 ). Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv Preprint arXiv:1408.5882 (2014)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501604.2501611"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 2016 AAAI Fall Symposium Series.","author":"Liu Fei","year":"2016","unstructured":"Fei Liu , Nicole Lee Fella , and Kexin Liao . 2016 a. Modeling language vagueness in privacy policies using deep neural networks . In Proceedings of the 2016 AAAI Fall Symposium Series. Fei Liu, Nicole Lee Fella, and Kexin Liao. 2016a. Modeling language vagueness in privacy policies using deep neural networks. In Proceedings of the 2016 AAAI Fall Symposium Series."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 25th International Conference on Computational Linguistics (COLING).","author":"Liu Fei","unstructured":"Fei Liu , Rohan Ramanath , Norman Sadeh , and Noah A. Smith . 2014. A step towards usable privacy policy: Automatic alignment of privacy statements . In Proceedings of the 25th International Conference on Computational Linguistics (COLING). Fei Liu, Rohan Ramanath, Norman Sadeh, and Noah A. Smith. 2014. A step towards usable privacy policy: Automatic alignment of privacy statements. In Proceedings of the 25th International Conference on Computational Linguistics (COLING)."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2016 AAAI Fall Symposium Series.","author":"Liu Frederick","year":"2016","unstructured":"Frederick Liu , Shomir Wilson , Florian Schaub , and Norman Sadeh . 2016 b. Analyzing vocabulary intersections of expert annotations and topic models for data practices in privacy policies . In Proceedings of the 2016 AAAI Fall Symposium Series. Frederick Liu, Shomir Wilson, Florian Schaub, and Norman Sadeh. 2016b. Analyzing vocabulary intersections of expert annotations and topic models for data practices in privacy policies. In Proceedings of the 2016 AAAI Fall Symposium Series."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481371"},{"key":"e_1_2_1_33_1","volume-title":"Retrieved","author":"Mahler Lars","year":"2015","unstructured":"Lars Mahler . 2015 . What Is NLP and Why Should Lawyers Care ? Retrieved March 12, 2018 from http:\/\/www.lawpracticetoday.org\/article\/nlp-lawyers\/. Lars Mahler. 2015. What Is NLP and Why Should Lawyers Care? Retrieved March 12, 2018 from http:\/\/www.lawpracticetoday.org\/article\/nlp-lawyers\/."},{"key":"e_1_2_1_34_1","volume-title":"The Stanford CoreNLP natural language processing toolkit","author":"Manning Christopher D.","unstructured":"Christopher D. Manning , Mihai Surdeanu , John Bauer , Jenny Finkel , Steven J. Bethard , and David McClosky . 2014. The Stanford CoreNLP natural language processing toolkit . In Association for Computational Linguistics (ACL) System Demonstrations . 55--60. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. 55--60."},{"key":"e_1_2_1_35_1","volume-title":"Browser Wars: A New Sequel? The Technology of Privacy","author":"McDonald Aleecia M.","year":"2013","unstructured":"Aleecia M. McDonald . 2013 . Browser Wars: A New Sequel? The Technology of Privacy . Silicon Flatirons Center , University of Colorado. Presented Jan. 11, 2013. Aleecia M. McDonald. 2013. Browser Wars: A New Sequel? The Technology of Privacy. Silicon Flatirons Center, University of Colorado. Presented Jan. 11, 2013."},{"key":"e_1_2_1_36_1","first-page":"540","article-title":"The cost of reading privacy policies","volume":"4","author":"McDonald Aleecia M.","year":"2008","unstructured":"Aleecia M. McDonald and Lorrie Faith Cranor . 2008 . The cost of reading privacy policies . I\/S: Journal of Law and Policy for the Information Society 4 , 3 (2008), 540 -- 561 . Aleecia M. McDonald and Lorrie Faith Cranor. 2008. The cost of reading privacy policies. I\/S: Journal of Law and Policy for the Information Society 4, 3 (2008), 540--561.","journal-title":"I\/S: Journal of Law and Policy for the Information Society"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39371-6_8"},{"key":"e_1_2_1_38_1","volume-title":"Semantic Processing of Legal Texts","author":"Montemagni Simonetta","unstructured":"Simonetta Montemagni , Wim Peters , and Daniela Tiscornia . 2010. Semantic Processing of Legal Texts . Springer . Simonetta Montemagni, Wim Peters, and Daniela Tiscornia. 2010. Semantic Processing of Legal Texts. Springer."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 670--679","author":"Negri Matteo","year":"2011","unstructured":"Matteo Negri , Luisa Bentivogli , Yashar Mehdad , Danilo Giampiccolo , and Alessandro Marchetti . 2011 . Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora . In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 670--679 . Matteo Negri, Luisa Bentivogli, Yashar Mehdad, Danilo Giampiccolo, and Alessandro Marchetti. 2011. Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 670--679."},{"key":"e_1_2_1_40_1","volume-title":"Retrieved","author":"Legislative Information Official California","year":"2003","unstructured":"Official California Legislative Information . 2003 . Online Privacy Protection Act of 2003 . Retrieved March 12, 2018 from http:\/\/leginfo.legislature.ca.gov\/faces\/billTextClient.xhtml?bill_id&equals;200320040AB68. Official California Legislative Information. 2003. Online Privacy Protection Act of 2003. Retrieved March 12, 2018 from http:\/\/leginfo.legislature.ca.gov\/faces\/billTextClient.xhtml?bill_id&equals;200320040AB68."},{"key":"e_1_2_1_41_1","volume-title":"PrivOnto: A semantic framework for the analysis of privacy policies. Semantic Web Journal Preprint","author":"Oltramari Alessandro","year":"2017","unstructured":"Alessandro Oltramari , Dhivya Piraviperumal , Florian Schaub , Shomir Wilson , Sushain Cherivirala , Thomas B. Norton , N. Cameron Russell , Peter Story , Joel Reidenberg , and Norman Sadeh . 2017. PrivOnto: A semantic framework for the analysis of privacy policies. Semantic Web Journal Preprint ( 2017 ), 1--19. Alessandro Oltramari, Dhivya Piraviperumal, Florian Schaub, Shomir Wilson, Sushain Cherivirala, Thomas B. Norton, N. Cameron Russell, Peter Story, Joel Reidenberg, and Norman Sadeh. 2017. PrivOnto: A semantic framework for the analysis of privacy policies. Semantic Web Journal Preprint (2017), 1--19."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL\u201914)","author":"Ramanath Rohan","unstructured":"Rohan Ramanath , Fei Liu , Norman Sadeh , and Noah A. Smith . 2014. Unsupervised alignment of privacy policies using hidden Markov models . In Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL\u201914) . ACL, 605--610. Rohan Ramanath, Fei Liu, Norman Sadeh, and Noah A. Smith. 2014. Unsupervised alignment of privacy policies using hidden Markov models. In Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL\u201914). ACL, 605--610."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916)","author":"Rao A.","year":"2016","unstructured":"A. Rao , F. Schaub , N. Sadeh , A. Acquisti , and R. Kang . 2016. Expecting the unexpected: Understanding mismatched privacy expectations online . In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916) . USENIX Association, 77--96. DOI:https:\/\/www.usenix.org\/system\/files\/conference\/soups 2016 \/soups2016-paper-rao.pdf. A. Rao, F. Schaub, N. Sadeh, A. Acquisti, and R. Kang. 2016. Expecting the unexpected: Understanding mismatched privacy expectations online. In Proceedings of the 12th Symposium on Usable Privacy and Security (SOUPS\u201916). USENIX Association, 77--96. DOI:https:\/\/www.usenix.org\/system\/files\/conference\/soups2016\/soups2016-paper-rao.pdf."},{"key":"e_1_2_1_46_1","article-title":"Automated comparisons of ambiguity in privacy policies and the impact of regulation","volume":"45","author":"Reidenberg Joel R.","year":"2016","unstructured":"Joel R. Reidenberg , Jaspreet Bhatia , Travis Breaux , and Thomas B. Norton . 2016 . Automated comparisons of ambiguity in privacy policies and the impact of regulation . Journal of Legal Studies 45 , 2 (15 Mar 2016), S163--S190. Joel R. Reidenberg, Jaspreet Bhatia, Travis Breaux, and Thomas B. Norton. 2016. Automated comparisons of ambiguity in privacy policies and the impact of regulation. Journal of Legal Studies 45, 2 (15 Mar 2016), S163--S190.","journal-title":"Journal of Legal Studies"},{"key":"e_1_2_1_47_1","first-page":"39","article-title":"Disagreeable privacy policies: Mismatches between meaning and users\u2019 understanding","volume":"30","author":"Reidenberg Joel R.","year":"2015","unstructured":"Joel R. Reidenberg , Travis Breaux , Lorrie Faith Cranor , Brian French , Amanda Grannis , James T. Graves , Fei Liu , Aleecia McDonald , Thomas B. Norton , Rohan Ramanath , N. Cameron Russell , Norman Sadeh , and Florian Schaub . 2015 a. Disagreeable privacy policies: Mismatches between meaning and users\u2019 understanding . Berkeley Tech. LJ 30 (2015), 39 . Joel R. Reidenberg, Travis Breaux, Lorrie Faith Cranor, Brian French, Amanda Grannis, James T. Graves, Fei Liu, Aleecia McDonald, Thomas B. Norton, Rohan Ramanath, N. Cameron Russell, Norman Sadeh, and Florian Schaub. 2015a. Disagreeable privacy policies: Mismatches between meaning and users\u2019 understanding. Berkeley Tech. LJ 30 (2015), 39.","journal-title":"Berkeley Tech. LJ"},{"key":"e_1_2_1_48_1","volume-title":"Norton","author":"Reidenberg Joel R.","year":"2015","unstructured":"Joel R. Reidenberg , N. Cameron Russell , Alexander J. Callen , Sophia Qasir , and Thomas B . Norton . 2015 b. Privacy harms and the effectiveness of the notice and choice framework. I\/S: Journal of Law 8 Policy for the Information Society 11 (2015). Joel R. Reidenberg, N. Cameron Russell, Alexander J. Callen, Sophia Qasir, and Thomas B. Norton. 2015b. Privacy harms and the effectiveness of the notice and choice framework. I\/S: Journal of Law 8 Policy for the Information Society 11 (2015)."},{"key":"e_1_2_1_49_1","volume-title":"Aleecia M. McDonald, Joel R. Reidenberg, Noah A. Smith, Fei Liu, N. Cameron Russell, Florian Schaub, and Shomir Wilson.","author":"Sadeh Norman","year":"2013","unstructured":"Norman Sadeh , Alessandro Acquisti , Travis D. Breaux , Lorrie Faith Cranor , Aleecia M. McDonald, Joel R. Reidenberg, Noah A. Smith, Fei Liu, N. Cameron Russell, Florian Schaub, and Shomir Wilson. 2013 . The Usable Privacy Policy Project: Combining Crowdsourcing , Machine Learning and Natural Language Processing to Semi-Automatically Answer Those Privacy Questions Users Care About. Tech. report CMU-ISR-13-119. Carnegie Mellon University . Norman Sadeh, Alessandro Acquisti, Travis D. Breaux, Lorrie Faith Cranor, Aleecia M. McDonald, Joel R. Reidenberg, Noah A. Smith, Fei Liu, N. Cameron Russell, Florian Schaub, and Shomir Wilson. 2013. The Usable Privacy Policy Project: Combining Crowdsourcing, Machine Learning and Natural Language Processing to Semi-Automatically Answer Those Privacy Questions Users Care About. Tech. report CMU-ISR-13-119. Carnegie Mellon University."},{"key":"e_1_2_1_50_1","volume-title":"Tech. Report CMU-LTI-17-005. Carnegie Mellon University.","author":"Sathyendra K.M.","year":"2017","unstructured":"K.M. Sathyendra , A. Ravichander , P. Story , A.W. Black , and N. Sadeh . 2017 a. Helping Users Understand Privacy Notices with Automated Question Answering Functionality: An Exploratory Study . Tech. Report CMU-LTI-17-005. Carnegie Mellon University. K.M. Sathyendra, A. Ravichander, P. Story, A.W. Black, and N. Sadeh. 2017a. Helping Users Understand Privacy Notices with Automated Question Answering Functionality: An Exploratory Study. Tech. Report CMU-LTI-17-005. Carnegie Mellon University."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1294"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2017.75"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 11th Symposium On Usable Privacy and Security (SOUPS","author":"Schaub Florian","year":"2015","unstructured":"Florian Schaub , Rebecca Balebako , Adam L. Durity , and Lorrie Faith Cranor . 2015 . A design space for effective privacy notices . In Proceedings of the 11th Symposium On Usable Privacy and Security (SOUPS 2015). USENIX Association, Ottawa, 1--17. Florian Schaub, Rebecca Balebako, Adam L. Durity, and Lorrie Faith Cranor. 2015. A design space for effective privacy notices. In Proceedings of the 11th Symposium On Usable Privacy and Security (SOUPS 2015). USENIX Association, Ottawa, 1--17."},{"key":"e_1_2_1_54_1","volume-title":"Crowdsourcing privacy policy analysis: Potential, challenges and best practices. it--Information Technology 58, 5","author":"Schaub Florian","year":"2016","unstructured":"Florian Schaub , Travis D. Breaux , and Norman Sadeh . 2016. Crowdsourcing privacy policy analysis: Potential, challenges and best practices. it--Information Technology 58, 5 ( 2016 ), 229--236. Florian Schaub, Travis D. Breaux, and Norman Sadeh. 2016. Crowdsourcing privacy policy analysis: Potential, challenges and best practices. it--Information Technology 58, 5 (2016), 229--236."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884855"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1621995.1622041"},{"key":"e_1_2_1_57_1","volume-title":"Retrieved","author":"DR.","year":"2012","unstructured":"Tos; DR. 2012 . Terms of Service Didn\u2019t Read. http:\/\/tosdr.org\/ . Retrieved March 12, 2018. Tos;DR. 2012. Terms of Service Didn\u2019t Read. http:\/\/tosdr.org\/. Retrieved March 12, 2018."},{"key":"e_1_2_1_58_1","unstructured":"University of Cambridge. 2013. Certificate of Proficiency in English (CPE) CEFR Level C2): Handbook for Teachers.  University of Cambridge. 2013. Certificate of Proficiency in English (CPE) CEFR Level C2): Handbook for Teachers."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of LREC 1st Workshop on Text Analytics for Cybersecurity and Online Safety (TA-COS\u201916)","author":"Wilson Shomir","year":"2016","unstructured":"Shomir Wilson , Florian Schaub , Aswarth Dara , Sushain K. Cherivirala , Sebastian Zimmeck , Mads Schaarup Andersen , Pedro Giovanni Leon , Eduard Hovy , and Norman Sadeh . 2016 a. Demystifying privacy policies with language technologies: Progress and challenges . In Proceedings of LREC 1st Workshop on Text Analytics for Cybersecurity and Online Safety (TA-COS\u201916) . ELRA, Portoro\u017e, Slovenia. Shomir Wilson, Florian Schaub, Aswarth Dara, Sushain K. Cherivirala, Sebastian Zimmeck, Mads Schaarup Andersen, Pedro Giovanni Leon, Eduard Hovy, and Norman Sadeh. 2016a. Demystifying privacy policies with language technologies: Progress and challenges. In Proceedings of LREC 1st Workshop on Text Analytics for Cybersecurity and Online Safety (TA-COS\u201916). ELRA, Portoro\u017e, Slovenia."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1126"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883035"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860485"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the USENIX Security Symposium.","author":"Zimmeck Sebastian","unstructured":"Sebastian Zimmeck and Steven M. Bellovin . 2014. Privee: An architecture for automatically analyzing web privacy policies . In Proceedings of the USENIX Security Symposium. Sebastian Zimmeck and Steven M. Bellovin. 2014. Privee: An architecture for automatically analyzing web privacy policies. In Proceedings of the USENIX Security Symposium."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2017.23034"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3230665","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3230665","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3230665","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:39:47Z","timestamp":1750210787000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3230665"}},"subtitle":["From Crowdsourcing to Automated Annotations"],"short-title":[],"issued":{"date-parts":[[2018,12,4]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,2,28]]}},"alternative-id":["10.1145\/3230665"],"URL":"https:\/\/doi.org\/10.1145\/3230665","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,12,4]]},"assertion":[{"value":"2017-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-12-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}