{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:11:18Z","timestamp":1784178678350,"version":"3.55.0"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,10,12]],"date-time":"2020-10-12T00:00:00Z","timestamp":1602460800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"EPSRC IAA"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,1,31]]},"abstract":"<jats:p>\n            Government documents must be manually reviewed to identify any\n            <jats:italic>sensitive<\/jats:italic>\n            information, e.g., confidential information, before being publicly archived. However, human-only sensitivity review is not practical for born-digital documents due to, for example, the volume of documents that are to be reviewed. In this work, we conduct a user study to evaluate the effectiveness of sensitivity classification for\n            <jats:italic>assisting<\/jats:italic>\n            human sensitivity reviewers. We evaluate how the accuracy and confidence levels of sensitivity classification affects the number of documents that are correctly judged as being sensitive (reviewer accuracy) and the time that it takes to sensitivity review a document (reviewing speed). In our within-subject study, the participants review government documents to identify real sensitivities while being assisted by three sensitivity classification\n            <jats:italic>treatments<\/jats:italic>\n            , namely\n            <jats:italic>None<\/jats:italic>\n            (no classification predictions),\n            <jats:italic>Medium<\/jats:italic>\n            (sensitivity predictions from a\n            <jats:italic>simulated<\/jats:italic>\n            classifier with a balanced accuracy (BAC) of 0.7), and\n            <jats:italic>Perfect<\/jats:italic>\n            (sensitivity predictions from a classifier with an accuracy of 1.0). Our results show that sensitivity classification leads to significant improvements (ANOVA,\n            <jats:italic>p<\/jats:italic>\n            &lt; 0.05) in reviewer accuracy in terms of BAC (+37.9%\n            <jats:italic>Medium<\/jats:italic>\n            , +60.0%\n            <jats:italic>Perfect<\/jats:italic>\n            ) and also in terms of F\n            <jats:sup>2<\/jats:sup>\n            (+40.8%\n            <jats:italic>Medium<\/jats:italic>\n            , +44.9%\n            <jats:italic>Perfect<\/jats:italic>\n            ). Moreover, we show that assisting reviewers with sensitivity classification predictions leads to significantly increased (ANOVA,\n            <jats:italic>p<\/jats:italic>\n            &lt; 0.05) mean reviewing speeds (+72.2%\n            <jats:italic>Medium<\/jats:italic>\n            , +61.6%\n            <jats:italic>Perfect<\/jats:italic>\n            ). We find that reviewers do not agree with the classifier significantly more as the classifier\u2019s confidence increases. However, reviewing speed is significantly increased when the reviewers agree with the classifier (ANOVA,\n            <jats:italic>p<\/jats:italic>\n            &lt; 0.05). Our in-depth analysis shows that when the reviewers are not assisted with sensitivity predictions, mean reviewing speeds are 40.5% slower for sensitive judgements compared to not-sensitive judgements. However, when the reviewers\n            <jats:italic>are<\/jats:italic>\n            assisted with sensitivity predictions, the difference in reviewing speeds between sensitive and not-sensitive judgements is reduced by \u02dc10%, from 40.5% to 30.8%. We also find that, for sensitive judgements, sensitivity classification predictions significantly increase mean reviewing speeds by 37.7% when the reviewers agree with the classifier\u2019s predictions (\n            <jats:italic>t<\/jats:italic>\n            -test,\n            <jats:italic>p<\/jats:italic>\n            &lt; 0.05). Overall, our findings demonstrate that sensitivity classification is a viable technology for assisting human reviewers with the sensitivity review of digital documents.\n          <\/jats:p>","DOI":"10.1145\/3417334","type":"journal-article","created":{"date-parts":[[2020,10,12]],"date-time":"2020-10-12T08:44:58Z","timestamp":1602492298000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["How the Accuracy and Confidence of Sensitivity Classification Affects Digital Sensitivity Review"],"prefix":"10.1145","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1266-5996","authenticated-orcid":false,"given":"Graham","family":"Mcdonald","sequence":"first","affiliation":[{"name":"University of Glasgow, Glasgow, Scotland, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Craig","family":"Macdonald","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, Scotland, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iadh","family":"Ounis","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, Scotland, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,12]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Sir Alex Allan. 2015. Government Digital Records and Archives Review. Cabinet Office. Retrieved from https:\/\/www.gov.uk\/government\/publications.  Sir Alex Allan. 2015. Government Digital Records and Archives Review. Cabinet Office. Retrieved from https:\/\/www.gov.uk\/government\/publications."},{"key":"e_1_2_1_2_1","unstructured":"Sir Alex Allan. 2014. Records Review. Cabinet Office. Retrieved from https:\/\/www.gov.uk\/government\/publications.  Sir Alex Allan. 2014. Records Review. Cabinet Office. Retrieved from https:\/\/www.gov.uk\/government\/publications."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (CAIR\u201918)","author":"Azzopardi Leif","year":"2018","unstructured":"Leif Azzopardi , Mateusz Dubiel , Martin Halvey , and Jeffery Dalton . 2018 . Conceptualizing agent-human interactions during the conversational search process . In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (CAIR\u201918) . Leif Azzopardi, Mateusz Dubiel, Martin Halvey, and Jeffery Dalton. 2018. Conceptualizing agent-human interactions during the conversational search process. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (CAIR\u201918)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950050"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806597"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348411"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.4.3.243"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 20th International Conference on Pattern Recognition. 3121--3124","author":"Brodersen Kay Henning","unstructured":"Kay Henning Brodersen , Cheng Soon Ong , Klaas Enno Stephan , and Joachim M. Buhmann . 2010. The balanced accuracy and its posterior distribution . In Proceedings of the 20th International Conference on Pattern Recognition. 3121--3124 . Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M. Buhmann. 2010. The balanced accuracy and its posterior distribution. In Proceedings of the 20th International Conference on Pattern Recognition. 3121--3124."},{"key":"e_1_2_1_9_1","volume-title":"Stanley","author":"Campbell Donald T.","year":"2015","unstructured":"Donald T. Campbell and Julian C . Stanley . 2015 . Experimental and Quasi-Experimental Designs for Research. Ravenio Books . Donald T. Campbell and Julian C. Stanley. 2015. Experimental and Quasi-Experimental Designs for Research. Ravenio Books."},{"key":"e_1_2_1_10_1","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"Cohen Jacob","unstructured":"Jacob Cohen . 1977. Statistical Power Analysis for the Behavioral Sciences . Academic Press . Jacob Cohen. 1977. Statistical Power Analysis for the Behavioral Sciences. Academic Press."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 5--14","author":"Gordon","unstructured":"Gordon V. Cormack and Maura R. Grossman. 2017. Navigating imprecision in relevance assessments on the road to total recall: Roger and me . In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 5--14 . Gordon V. Cormack and Maura R. Grossman. 2017. Navigating imprecision in relevance assessments on the road to total recall: Roger and me. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 5--14."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 19th Text REtrieval Conference.","author":"Cormack Gordon V.","unstructured":"Gordon V. Cormack , Maura R. Grossman , Bruce Hedin , and Douglas W. Oard . 2010. Overview of the TREC 2010 Legal Track . In Proceedings of the 19th Text REtrieval Conference. Gordon V. Cormack, Maura R. Grossman, Bruce Hedin, and Douglas W. Oard. 2010. Overview of the TREC 2010 Legal Track. In Proceedings of the 19th Text REtrieval Conference."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.20982\/tqmp.01.1.p042"},{"key":"e_1_2_1_14_1","first-page":"217","article-title":"Confidence intervals and replication: Where will the next mean fall?Psychol","volume":"11","author":"Cumming Geoff","year":"2006","unstructured":"Geoff Cumming and Robert Maillardet . 2006 . Confidence intervals and replication: Where will the next mean fall?Psychol . Methods 11 , 3 (2006), 217 . Geoff Cumming and Robert Maillardet. 2006. Confidence intervals and replication: Where will the next mean fall?Psychol. Methods 11, 3 (2006), 217.","journal-title":"Methods"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 21st Australasian Document Computing Symposium. 41--48","author":"Damessie Tadele T.","unstructured":"Tadele T. Damessie , Falk Scholer , and J. Shane Culpepper . 2016. The influence of topic difficulty, relevance level, and document ordering on relevance judging . In Proceedings of the 21st Australasian Document Computing Symposium. 41--48 . Tadele T. Damessie, Falk Scholer, and J. Shane Culpepper. 2016. The influence of topic difficulty, relevance level, and document ordering on relevance judging. In Proceedings of the 21st Australasian Document Computing Symposium. 41--48."},{"key":"e_1_2_1_16_1","unstructured":"DARPA. 2010. DARPA New Technologies to Support Declassification. (2010). Retrieved from http:\/\/fas.org\/sgp\/news\/2010\/09\/darpa-declass.pdf.  DARPA. 2010. DARPA New Technologies to Support Declassification. (2010). Retrieved from http:\/\/fas.org\/sgp\/news\/2010\/09\/darpa-declass.pdf."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_2_1_18_1","unstructured":"Freedom of Information Act 2000. c. 36. Retrieved from https:\/\/www.legislation.gov.uk\/ukpga\/2000\/36\/contents.  Freedom of Information Act 2000. c. 36. Retrieved from https:\/\/www.legislation.gov.uk\/ukpga\/2000\/36\/contents."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1749603.1749605"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02289823"},{"key":"e_1_2_1_21_1","first-page":"1","article-title":"Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review","volume":"17","author":"Grossman Maura R.","year":"2010","unstructured":"Maura R. Grossman and Gordon V. Cormack . 2010 . Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review . Rich. J. Law Technol. 17 , 3 (2010), 1 . Maura R. Grossman and Gordon V. Cormack. 2010. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Rich. J. Law Technol. 17, 3 (2010), 1.","journal-title":"Rich. J. Law Technol."},{"key":"e_1_2_1_22_1","unstructured":"Victoria Lain. 2013. Digital Records Sensitivity Review. Retrieved from https:\/\/blog.nationalarchives.gov.uk\/blog\/digital-records-sensitivity-review.  Victoria Lain. 2013. Digital Records Sensitivity Review. Retrieved from https:\/\/blog.nationalarchives.gov.uk\/blog\/digital-records-sensitivity-review."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03210951"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3223045"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080731"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809496"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_35"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_65"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298962"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-06028-6_48"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0003-6870(00)00071-5"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 27th Annual Conference on Neural Information Processing Systems. 3111--3119","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov , Ilya Sutskever , Kai Chen , Gregory S. Corrado , and Jeffrey Dean . 2013 . Distributed representations of words and phrases and their compositionality . In Proceedings of the 27th Annual Conference on Neural Information Processing Systems. 3111--3119 . Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems. 3111--3119."},{"key":"e_1_2_1_33_1","first-page":"61","article-title":"Confidence intervals from normalized data: A correction to cousineau (2005)","volume":"4","author":"Morey Richard D.","year":"2008","unstructured":"Richard D. Morey . 2008 . Confidence intervals from normalized data: A correction to cousineau (2005) . Reason 4 , 2 (2008), 61 -- 64 . Richard D. Morey. 2008. Confidence intervals from normalized data: A correction to cousineau (2005). Reason 4, 2 (2008), 61--64.","journal-title":"Reason"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-010-9093-9"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000025"},{"key":"e_1_2_1_36_1","unstructured":"Public Records Act 1958. c. 51. Retrieved from http:\/\/www.legislation.gov.uk\/ukpga\/Eliz2\/6-7\/51.  Public Records Act 1958. c. 51. Retrieved from http:\/\/www.legislation.gov.uk\/ukpga\/Eliz2\/6-7\/51."},{"key":"e_1_2_1_37_1","first-page":"2","article-title":"FACTS-IR: Fairness, accountability, confidentiality, transparency, and safety in information retrieval","volume":"53","author":"Roegiest Adam","year":"2019","unstructured":"Adam Roegiest , Aldo Lipani , Alex Beutel , Alexandra Olteanu , Ana Lucic , Ana-Andreea Stoica , Anubrata Das , Asia Biega , Bart Voorn , Claudia Hauff , Damiano Spina , David Lewis , Douglas W. Oard , Emine Yilmaz , Faegheh Hasibi , Gabriella Kazai , Graham McDonald , Hinda Haned , Iadh Ounis , Ilse van der Linden , Jean Garcia-Gathright , Joris Baan , Kamuela N. Lau , Krisztian Balog , Maarten de Rijke , Mahmoud Sayed , Maria Panteli , Mark Sanderson , Matthew Lease , Michael D. Ekstrand , Preethi Lahoti , and Toshihiro Kamishima . 2019 . FACTS-IR: Fairness, accountability, confidentiality, transparency, and safety in information retrieval . SIGIR For. 53 , 2 (December 2019). Adam Roegiest, Aldo Lipani, Alex Beutel, Alexandra Olteanu, Ana Lucic, Ana-Andreea Stoica, Anubrata Das, Asia Biega, Bart Voorn, Claudia Hauff, Damiano Spina, David Lewis, Douglas W. Oard, Emine Yilmaz, Faegheh Hasibi, Gabriella Kazai, Graham McDonald, Hinda Haned, Iadh Ounis, Ilse van der Linden, Jean Garcia-Gathright, Joris Baan, Kamuela N. Lau, Krisztian Balog, Maarten de Rijke, Mahmoud Sayed, Maria Panteli, Mark Sanderson, Matthew Lease, Michael D. Ekstrand, Preethi Lahoti, and Toshihiro Kamishima. 2019. FACTS-IR: Fairness, accountability, confidentiality, transparency, and safety in information retrieval. SIGIR For. 53, 2 (December 2019).","journal-title":"SIGIR For."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23363"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 615--624","author":"Mahmoud","unstructured":"Mahmoud F. Sayed and Douglas W. Oard. 2019. Jointly modeling relevance and sensitivity for search among sensitive content . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 615--624 . Mahmoud F. Sayed and Douglas W. Oard. 2019. Jointly modeling relevance and sensitivity for search among sensitive content. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 615--624."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"key":"e_1_2_1_41_1","volume-title":"Rohan Shah, and Matthew Connelly.","author":"Souza Renato Rocha","year":"2016","unstructured":"Renato Rocha Souza , Flavio Codeco Coelho , Rohan Shah, and Matthew Connelly. 2016 . Using artificial intelligence to identify state secrets. Arxiv:1611.00356 (2016) Retrieved from https:\/\/arxiv.org\/abs\/1611.00356. Renato Rocha Souza, Flavio Codeco Coelho, Rohan Shah, and Matthew Connelly. 2016. Using artificial intelligence to identify state secrets. Arxiv:1611.00356 (2016) Retrieved from https:\/\/arxiv.org\/abs\/1611.00356."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218488502001648"},{"key":"e_1_2_1_43_1","unstructured":"The National Archives. 2016. The Application of Technology-assisted Review to Born-digital Records Transfer Inquiries and Beyond. The National Archives. Retrieved from http:\/\/www.nationalarchives.gov.uk\/documents\/technology-assisted-review-to-born-digital-records-transfer.pdf.  The National Archives. 2016. The Application of Technology-assisted Review to Born-digital Records Transfer Inquiries and Beyond. The National Archives. Retrieved from http:\/\/www.nationalarchives.gov.uk\/documents\/technology-assisted-review-to-born-digital-records-transfer.pdf."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/1059502.1059508"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Alistair G. Tough. 2018. The scope and appetite for technology-assisted sensitivity reviewing of born-digital records in a resource poor environment: A case study from malawi. In Handbook of Research on Heritage Management and Preservation. IGI Global 175--182.  Alistair G. Tough. 2018. The scope and appetite for technology-assisted sensitivity reviewing of born-digital records in a resource poor environment: A case study from malawi. In Handbook of Research on Heritage Management and Preservation. IGI Global 175--182.","DOI":"10.4018\/978-1-5225-3137-1.ch009"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148176"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_2_1_48_1","volume-title":"Statistical Principles in Experimental Design","author":"Winer Ben James","unstructured":"Ben James Winer . 1962. Statistical Principles in Experimental Design . McGraw-Hill , New York, NY . Ben James Winer. 1962. Statistical Principles in Experimental Design. McGraw-Hill, New York, NY."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3417334","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3417334","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:14Z","timestamp":1750193294000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3417334"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,12]]},"references-count":48,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1,31]]}},"alternative-id":["10.1145\/3417334"],"URL":"https:\/\/doi.org\/10.1145\/3417334","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,12]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}