{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T16:53:35Z","timestamp":1771606415804,"version":"3.50.1"},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,12,10]],"date-time":"2023-12-10T00:00:00Z","timestamp":1702166400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,12,10]],"date-time":"2023-12-10T00:00:00Z","timestamp":1702166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"The MITRE Corporation","award":["IRD300.23.D1.0CA"],"award-info":[{"award-number":["IRD300.23.D1.0CA"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Law"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Freedom of information laws promote transparency by permitting individuals and organizations to obtain government documents. However, exemptions from disclosure are necessary to protect privacy and to permit government officials to deliberate freely. Deliberative language is often the most challenging and burdensome exemption to detect, leading to high processing costs and delays in responding to open-records requests. This paper describes a novel deliberative-language detection model trained on a new annotated training set. The deliberative-language detection model is a component of a decision-support system for open-records requests under the US Freedom of Information Act, the <jats:italic>FOIA Assistant<\/jats:italic>, that ingests documents responsive to an open-records requests, suggests passages likely to be subject to deliberative language, privacy, or other exemptions, and assists analysts in rapidly redacting suggested passages. The tool\u2019s interface is based on extensive human-factors and usability studies with analysts and is currently in operational testing by multiple US federal agencies.<\/jats:p>","DOI":"10.1007\/s10506-023-09383-6","type":"journal-article","created":{"date-parts":[[2023,12,10]],"date-time":"2023-12-10T10:01:15Z","timestamp":1702202475000},"page":"171-197","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Decision support for detecting sensitive text in government records"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9362-495X","authenticated-orcid":false,"given":"Karl","family":"Branting","sequence":"first","affiliation":[]},{"given":"Bradford","family":"Brown","sequence":"additional","affiliation":[]},{"given":"Chris","family":"Giannella","sequence":"additional","affiliation":[]},{"given":"James Van","family":"Guilder","sequence":"additional","affiliation":[]},{"given":"Jeff","family":"Harrold","sequence":"additional","affiliation":[]},{"given":"Sarah","family":"Howell","sequence":"additional","affiliation":[]},{"given":"Jason R.","family":"Baron","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,12,10]]},"reference":[{"key":"9383_CR1","unstructured":"Apache PDFBox 2.0.27 (2023) https:\/\/pdfbox.apache.org\/. Accessed 30 January 2023"},{"key":"9383_CR2","unstructured":"Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the seventh international conference on language resources and evaluation (LREC\u201910). European Language Resources Association (ELRA), Valletta, Malta"},{"issue":"1","key":"9383_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3481045","volume":"15","author":"JR Baron","year":"2022","unstructured":"Baron JR, Sayed MF, Oard DW (2022) Providing more efficient access to government records: a use case involving application of machine learning to improve FOIA review for the deliberative process privilege. J Comput Cult Herit 15(1):1\u201319","journal-title":"J Comput Cult Herit"},{"key":"9383_CR4","unstructured":"Canada, Access to Information Act (1995) \u00a7\u00a714(a), 15(1), 21(1)(b) & 69(1)(c)"},{"key":"9383_CR5","doi-asserted-by":"publisher","unstructured":"Chhatwal R, Keeling R, Gronvall P, Huber-Fliflet N, Zhang J, Zhao H (2020) CNN application in detection of privileged documents in legal document review. In: 2020 IEEE international conference on big data (big data), pp 1485\u20131492. https:\/\/doi.org\/10.1109\/BigData50022.2020.9378182","DOI":"10.1109\/BigData50022.2020.9378182"},{"issue":"1","key":"9383_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G (2020) The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21(1):1\u201313","journal-title":"BMC Genomics"},{"key":"9383_CR7","unstructured":"Clinton Digital Library (2023) https:\/\/clinton.presidentiallibraries.us\/. Accessed 1 December 2023"},{"key":"9383_CR8","volume-title":"Practical nonparametric statistics","author":"WJ Conover","year":"1999","unstructured":"Conover WJ (1999) Practical nonparametric statistics, 3rd edn. Wiley, New York","edition":"3"},{"key":"9383_CR9","unstructured":"Consumers\u2019 Checkbook v.HHS (2009) For the study of Servs. v. HHS. 554 F.2d 1046, 1057 (D.C. Cir. 2009) (FOIA\u2019s \u201cpresumption favoring disclosure ... is at its zenith under Exemption 6\u201d)"},{"key":"9383_CR10","unstructured":"Cornell Movie Review Data (2023) https:\/\/www.cs.cornell.edu\/people\/pabo\/movie-review-data\/. Accessed 31 January 2023"},{"key":"9383_CR11","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL. Association for Computational Linguistics, pp 4171\u20134186"},{"key":"9383_CR12","unstructured":"Electronic Freedom of Information Act Amendments of 1996 (1996) Pub. L. 104-231, amending 5 U.S.C. 552(f)(2)(a) (defining \u201crecord\u201d as including in an \u201celectronic format\u201d)"},{"key":"9383_CR13","unstructured":"FBI v. Abramson (1982) 465 U.S. 615, 630"},{"key":"9383_CR14","unstructured":"Freedom of Information Act (1966) Pub. L. 89-487, codified at 5 U.S.C. 552(b)"},{"key":"9383_CR15","unstructured":"Fractalego (2020) Subjectivity detection with Bi-RNNs. https:\/\/github.com\/fractalego\/subjectivity_classifier"},{"key":"9383_CR16","unstructured":"Freedom of Information Act as  amended (2023) 5 U.S.C. 552(b)(1)-(9) and (c)(1)-(3). Accessed 1 December 2023"},{"key":"9383_CR17","doi-asserted-by":"crossref","unstructured":"Heffernan v. Azar (2018) 317 F.Supp.3d 94, 125 (D.D.C.2018)","DOI":"10.1007\/s00712-018-0632-2"},{"key":"9383_CR18","unstructured":"India Right to Information Act (2005) \u00a78, Clause I"},{"key":"9383_CR19","unstructured":"International FOI Laws (2023) National Freedom of Information Coalition. https:\/\/www.nfoic.org\/international-foi-laws\/. Accessed 30 November 2023"},{"issue":"CSCW1","key":"9383_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3449207","volume":"5","author":"M Iqbal","year":"2021","unstructured":"Iqbal M, Shilton K, Sayed M, Oard D, Rivera J, Cox W (2021) Search with discretion: value sensitive design of training data for information retrieval. Proc ACM Hum\u2013Comput Interact 5(CSCW1):1\u201320","journal-title":"Proc ACM Hum\u2013Comput Interact"},{"key":"9383_CR21","unstructured":"Israel Freedom of Information Act (1998) \u00a79.2"},{"key":"9383_CR22","unstructured":"Judicial Watch v. State Department (2018) 349 F.Supp.3d 1, 7 (D.D.C. 2018)"},{"key":"9383_CR23","unstructured":"Keras (2023) https:\/\/keras.io\/. Accessed 31 January 2023"},{"issue":"4","key":"9383_CR24","doi-asserted-by":"publisher","first-page":"150","DOI":"10.3390\/info10040150","volume":"10","author":"K Kowsari","year":"2019","unstructured":"Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Information 10(4):150","journal-title":"Information"},{"issue":"2","key":"9383_CR25","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","volume":"405","author":"BW Matthews","year":"1975","unstructured":"Matthews BW (1975) Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct 405(2):442\u2013451","journal-title":"Biochim Biophys Acta (BBA) Protein Struct"},{"key":"9383_CR26","doi-asserted-by":"crossref","unstructured":"McDonald G, Macdonald C, Ounis I (2017) Enhancing sensitivity classification with semantic features using word embeddings. In: Jose, J.M., Hauff, C., Alting\u00f6vde, I.S., Song, D., Albakour, D., Watt, S.N.K., Tait, J. (eds.) Advances in information retrieval\u201439th European conference on IR Research, ECIR 2017, Aberdeen, UK, April 8\u201313, 2017, proceedings, lecture notes in computer science, vol 10193, pp 450\u2013463","DOI":"10.1007\/978-3-319-56608-5_35"},{"issue":"1","key":"9383_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3417334","volume":"39","author":"G McDonald","year":"2020","unstructured":"McDonald G, Macdonald C, Ounis I (2020) How sensitivity classification effectiveness impacts reviewers in technology-assisted sensitivity review. ACM Trans Inf Syst 39(1):1\u201334","journal-title":"ACM Trans Inf Syst"},{"key":"9383_CR28","doi-asserted-by":"crossref","unstructured":"McDonald G, Macdonald C, Ounis I, Gollins T (2014) Towards a classifier for digital sensitivity review. In: de Rijke M, Kenter T, de Vries AP, Zhai C, de\u00a0Jong F, Radinsky K, Hofmann K (eds) Advances in information retrieval\u201436th European conference on IR Research, ECIR 2014. Amsterdam, The Netherlands, April 13\u201316, 2014. proceedings, lecture notes in computer science, vol 8416. Springer, pp. 500\u2013506","DOI":"10.1007\/978-3-319-06028-6_48"},{"issue":"3","key":"9383_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3439726","volume":"54","author":"S Minaee","year":"2021","unstructured":"Minaee S, Kalchbrenner N, Cambria E, Nikzad N, Chenaghlu M, Gao J (2021) Deep learning-based text classification: a comprehensive review. ACM Comput Surv 54(3):1\u201340","journal-title":"ACM Comput Surv"},{"key":"9383_CR30","unstructured":"Mosbach M, Andriushchenko M, Klakow D (2021) On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In: Proceedings of the 9th international conference on learning representations, ICLR"},{"key":"9383_CR31","doi-asserted-by":"crossref","unstructured":"NARA v. Favish (2004) 541 U.S. 157, 165\u201366","DOI":"10.1002\/scin.5591651014"},{"key":"9383_CR32","doi-asserted-by":"crossref","unstructured":"Narvala H, Mcdonald G, Ounis I (2022) The role of latent semantic categories and clustering in enhancing the efficiency of human sensitivity review. In: ACM SIGIR conference on human information interaction and retrieval, pp 56\u201366","DOI":"10.1145\/3498366.3505824"},{"key":"9383_CR33","unstructured":"NLRB v. Sears (1975) Roebuck and Co. 421 U.S. 132, 150"},{"issue":"5","key":"9383_CR34","first-page":"92","volume":"11","author":"V Nayak","year":"2019","unstructured":"Nayak V, D\u2019Souza R (2019) Comparison of multi-criteria decision making methods used in requirement engineering. CiiT Int J Artif Intell Syst Mach Learn 11(5):92\u201396","journal-title":"CiiT Int J Artif Intell Syst Mach Learn"},{"key":"9383_CR35","unstructured":"NIST Guide to Protecting the Confidentiality of Personally Identifiable Information (PII) (2010) Special Publication 800-122. https:\/\/nvlpubs.nist.gov\/nistpubs\/legacy\/sp\/nistspecialpublication800-122.pdf"},{"key":"9383_CR36","doi-asserted-by":"crossref","unstructured":"Pearson C, Seliya N, Dave R (2021) Named entity recognition in unstructured medical text documents. In: 2021 International conference on electrical, computer and energy technologies (ICECET), pp 1\u20136","DOI":"10.1109\/ICECET52533.2021.9698694"},{"key":"9383_CR37","unstructured":"PDF in Government (2023) https:\/\/www.talkingpdf.org\/pdf-in-government\/. Accessed 4 February 2023. (\u201cThere\u2019s not a federal agency that does not use PDF,\u201d says Greg Pisocky, Business Development Manager for Adobe Systems. \u2018Acrobat software and Adobe PDF are key technologies in some capacity at all branches and levels of government, the military and virtually every agency\u2019)"},{"issue":"3","key":"9383_CR38","first-page":"69","volume":"8","author":"C Righi","year":"2013","unstructured":"Righi C, James J, Beasley M, Day DL, Fox JE, Gieber J, Howe C, Ruby L (2013) Card sort analysis best practices. J Usability Stud 8(3):69\u201389","journal-title":"J Usability Stud"},{"issue":"5","key":"9383_CR39","first-page":"507","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG (2010) Mayo clinical text analysis and knowledge extraction system (cTAKES). JAMIA 17(5):507\u2013513","journal-title":"JAMIA"},{"key":"9383_CR40","doi-asserted-by":"crossref","unstructured":"Sayed MF, Oard DW (2019) Jointly Modeling Relevance and Sensitivity for Search among Sensitive Content. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, SIGIR. Association for Computing Machinery, pp 615\u2013624","DOI":"10.1145\/3331184.3331256"},{"key":"9383_CR41","first-page":"3405","volume":"1","author":"E Sheng","year":"2019","unstructured":"Sheng E, Chang KW, Natarajan P, Peng N (2019) The woman worked as a babysitter: on biases in language generation. EMNLP\/IJCNLP 1:3405\u20133410","journal-title":"EMNLP\/IJCNLP"},{"key":"9383_CR42","unstructured":"South Africa, Draft Model Freedom of Information Law (Adopted in 1999) \u00a725(1)(a)-(d)"},{"key":"9383_CR43","unstructured":"SpaCy (2023) Industrial-strength natural language process in python. https:\/\/spacy.io\/. Accessed 19 January 2023"},{"key":"9383_CR44","unstructured":"State Freedom of Information Laws (2023) National Freedom of Information Coalition. https:\/\/www.nfoic.org\/state-freedom-of-information-laws\/. Accessed 4 February 2023"},{"key":"9383_CR45","unstructured":"Summary of Annual FOIA Reports for Fiscal Year (2022) US Department of Justice. https:\/\/www.justice.gov\/oip\/page\/file\/1581856\/download. Accessed 14 May 2023"},{"key":"9383_CR46","unstructured":"UK Freedom of Information Act (2000) \u00a735(1a)-(1d). See generally, information commission\u2019s office, government policy (section 35). https:\/\/ico.org.uk\/media\/for-organisations\/documents\/1200\/government-policy-foi-section-35-guidance.pdf. Accessed 4 February 2023"},{"key":"9383_CR47","unstructured":"US Department of State v. Washington Post Co (1982) 456 U.S. 595, 602"},{"key":"9383_CR48","unstructured":"U.S. Fish and Wildlife Service v. Sierra Club Inc (2021) 141 Supreme Court 777"},{"key":"9383_CR49","doi-asserted-by":"crossref","unstructured":"Washington Post Co. v. HHS (1982) 690 F.2d 252, 261 (D.C. Cir. 1982)","DOI":"10.1088\/0031-8949\/1982\/T2A\/032"},{"key":"9383_CR50","unstructured":"Wikipedia: Freedom of Information Laws by Country (2023) https:\/\/en.wikipedia.org\/wiki\/Freedom_of_information_laws_by_country. Accessed 1 February 2023"}],"container-title":["Artificial Intelligence and Law"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10506-023-09383-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10506-023-09383-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10506-023-09383-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,5]],"date-time":"2025-03-05T08:15:34Z","timestamp":1741162534000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10506-023-09383-6"}},"subtitle":["Anonymous submission"],"short-title":[],"issued":{"date-parts":[[2023,12,10]]},"references-count":50,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["9383"],"URL":"https:\/\/doi.org\/10.1007\/s10506-023-09383-6","relation":{},"ISSN":["0924-8463","1572-8382"],"issn-type":[{"value":"0924-8463","type":"print"},{"value":"1572-8382","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,10]]},"assertion":[{"value":"7 November 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 December 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest. There is and has been no financial relationship between any author and any organization of relevance to this work. Further, no author is currently in any negotiations regarding future paid employment with any organization of relevance. The manuscript has not been submitted or published anywhere else, nor will it be submitted elsewhere until completion of the editorial process. All authors have approved the manuscript for submission and consent to publication should this submission be successful. All interviews and workshops were conducted in accordance with procedures approved by the the authors\u2019 Institutional Review Board.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}