{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T16:59:11Z","timestamp":1776790751931,"version":"3.51.2"},"reference-count":29,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2025,2,12]],"date-time":"2025-02-12T00:00:00Z","timestamp":1739318400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Digit. Gov.: Res. Pract."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>\n            This article introduces Instru\u00e7\u00e3o Assistida com Intelig\u00eancia Artificial (INACIA), a groundbreaking system designed to integrate Large Language Models (LLMs) into the operational framework of Brazilian Federal Court of Accounts. The system automates various stages of case analysis, including basic information extraction, admissibility examination,\n            <jats:italic>Periculum in mora<\/jats:italic>\n            and\n            <jats:italic>Fumus boni iuris<\/jats:italic>\n            analyses, and recommendations generation. Through a series of experiments, we demonstrate INACIA\u2019s potential in extracting relevant information from case documents, evaluating its legal plausibility, and formulating propositions for judicial decision-making. Utilizing a validation dataset alongside LLMs, our evaluation methodology, to the best of our knowledge, presents a novel approach to assessing system performance, correlating highly with human judgment. These results underscore INACIA\u2019s potential in complex legal task handling while also acknowledging the current limitations. This study discusses possible improvements and the broader implications of applying Artificial Intelligence (AI) in legal contexts, suggesting that INACIA represents a significant step toward integrating AI in legal systems globally, albeit with cautious optimism grounded in the empirical findings.\n          <\/jats:p>","DOI":"10.1145\/3652951","type":"journal-article","created":{"date-parts":[[2024,3,20]],"date-time":"2024-03-20T12:06:57Z","timestamp":1710936417000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5478-438X","authenticated-orcid":false,"given":"Jayr","family":"Pereira","sequence":"first","affiliation":[{"name":"NeuralMind, Campinas, Brazil"},{"name":"University of Campinas, Campinas, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1669-074X","authenticated-orcid":false,"given":"Andre","family":"Assumpcao","sequence":"additional","affiliation":[{"name":"National Center for State Courts (NCSC), Williamsburg, United States"},{"name":"Brazilian Association of Jurimetrics (ABJ), S\u00e3o Paulo, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1680-6389","authenticated-orcid":false,"given":"Julio","family":"Trecenti","sequence":"additional","affiliation":[{"name":"Terranova Consultoria, S\u00e3o Paulo, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6364-9592","authenticated-orcid":false,"given":"Luiz","family":"Airosa","sequence":"additional","affiliation":[{"name":"Brazilian Federal Court of Accounts (TCU), Bras\u00edlia Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8473-069X","authenticated-orcid":false,"given":"Caio","family":"Lente","sequence":"additional","affiliation":[{"name":"Terranova Consultoria, S\u00e3o Paulo, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2620-0768","authenticated-orcid":false,"given":"Jhonatan","family":"Cl\u00e9to","sequence":"additional","affiliation":[{"name":"NeuralMind.ai, Campinas Brazil"},{"name":"University of Campinas, Campinas, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4336-2143","authenticated-orcid":false,"given":"Guilherme","family":"Dobins","sequence":"additional","affiliation":[{"name":"NeuralMind.ai, Campinas Brazil"},{"name":"University of Campinas, Campinas, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2600-6035","authenticated-orcid":false,"given":"Rodrigo","family":"Nogueira","sequence":"additional","affiliation":[{"name":"NeuralMind.ai, Campinas Brazil"},{"name":"University of Campinas, Campinas, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9345-8292","authenticated-orcid":false,"given":"Luis","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Brazilian Federal Court of Accounts (TCU), Bras\u00edlia Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5652-0852","authenticated-orcid":false,"given":"Roberto","family":"Lotufo","sequence":"additional","affiliation":[{"name":"NeuralMind.ai, Campinas Brazil"},{"name":"University of Campinas, Campinas, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,12]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","unstructured":"Daron Acemoglu and David Autor. 2011. Skills tasks and technologies: Implications for employment and earnings. In Handbook of Labor Economics Vol. 4. Elsevier 1043\u20131171. 10.1016\/S0169-7218(11)02410-5","DOI":"10.1016\/S0169-7218(11)02410-5"},{"key":"e_1_3_3_3_2","volume-title":"Proceedings of the 3rd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES \u201922)","author":"Almeida Thales Sales","year":"2022","unstructured":"Thales Sales Almeida, Thiago Laitz, Jo\u00e3o Ser\u00f3dio, Luiz Henrique Bonifacio, Roberto Lotufo, and Rodrigo Nogueira. 2022. NeuralSearchX: Serving a multi-billion-parameter reranker for multilingual metasearch at a low cost. In Proceedings of the 3rd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES \u201922)."},{"key":"e_1_3_3_4_2","unstructured":"Apache. [n.d.]. Apache Tika. Retrieved December 11 2023 from https:\/\/tika.apache.org\/"},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","unstructured":"Ilias Chalkidis Manos Fergadiotis Prodromos Malakasiotis Nikolaos Aletras and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. arXiv:2010.02559. Retrieved from https:\/\/arxiv.org\/abs\/2010.02559","DOI":"10.18653\/v1\/2020.findings-emnlp.261"},{"key":"e_1_3_3_6_2","unstructured":"Liting Chen Lu Wang Hang Dong Yali Du Jie Yan Fangkai Yang Shuang Li Pu Zhao Si Qin Saravan Rajmohan Qingwei Lin and Dongmei Zhang. 2023. Introspective tips: Large language model for in-context decision making. arxiv:2305.11598 [cs.AI]. Retrieved from https:\/\/arxiv.org\/abs\/2305.11598"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00373"},{"key":"e_1_3_3_8_2","unstructured":"Mathieu Fenniak. [n.d.]. PyPDF2. Retrieved December 11 2023 from https:\/\/pypdf2.readthedocs.io\/en\/3.0.0\/"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1162\/qjec.2008.123.2.703"},{"key":"e_1_3_3_10_2","unstructured":"Jinlan Fu See-Kiong Ng Zhengbao Jiang and Pengfei Liu. 2023. GPTScore: Evaluate as you desire. arxiv:2302.04166 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2302.04166"},{"key":"e_1_3_3_11_2","unstructured":"Google. [n.d.]. Google Cloud Vision API. Retrieved December 11 2023 from https:\/\/cloud.google.com\/vision\/"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3573128.3609344"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nlp.2023.100048"},{"key":"e_1_3_3_14_2","unstructured":"Takeshi Kojima Shixiang Shane Gu Machel Reid Yutaka Matsuo and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv:2205.11916. Retrieved from https:\/\/arxiv.org\/abs\/2205.11916"},{"key":"e_1_3_3_15_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et\u00a0al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, Vol. 33, 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","unstructured":"Yang Liu Dan Iter Yichong Xu Shuohang Wang Ruochen Xu and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. arxiv:2303.16634 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2303.16634","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"e_1_3_3_17_2","unstructured":"Microsoft. [n.d.]. Microsoft Azure Form Recognizer. Retrieved December 12 2023 from https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services\/form-recognizer\/"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.63"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3594536.3595165"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-28238-6_44"},{"key":"e_1_3_3_21_2","unstructured":"Thomson Reuters Institute. 2024. Generative AI and the Courts: Balancing Efficiency and Legal Obligations. Retrieved February 9 2024 from https:\/\/www.thomsonreuters.com\/en-us\/posts\/government\/generative-ai-courts\/"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-225.routing-city"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604570"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3594536.3595161"},{"key":"e_1_3_3_25_2","unstructured":"Murray Shanahan and Catherine Clarke. 2023. Evaluating large language model creativity from a literary perspective. arxiv:2312.03746 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2312.03746"},{"issue":"1","key":"e_1_3_3_26_2","first-page":"184","article-title":"From byte to bail: Assessing challenges and opportunities in AI-driven criminal justice systems","volume":"4","author":"Sipra Sohail Zafar","year":"2024","unstructured":"Sohail Zafar Sipra, Muhammad Ihsan Ilahi, and Muhammad Imran Khan. 2024. From byte to bail: Assessing challenges and opportunities in AI-driven criminal justice systems. Int. J. Hum. Soc. 4, 1 (2024), 184\u2013195.","journal-title":"Int. J. Hum. Soc."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2007.4376991"},{"key":"e_1_3_3_28_2","first-page":"127","article-title":"Auditing institutions","author":"Speck Bruno W.","year":"2011","unstructured":"Bruno W. Speck. 2011. Auditing institutions. Corruption and Democracy in Brazil: the Struggle for Accountability. 127\u2013161.","journal-title":"Corruption and Democracy in Brazil: the Struggle for Accountability"},{"key":"e_1_3_3_29_2","unstructured":"Zhongxiang Sun. 2023. A short survey of viewing large language models in legal aspect. arxiv:2303.09136 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2303.09136"},{"key":"e_1_3_3_30_2","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. arxiv:2210.03629 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2210.03629"}],"container-title":["Digital Government: Research and Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652951","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3652951","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:58Z","timestamp":1750295878000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652951"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,12]]},"references-count":29,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3652951"],"URL":"https:\/\/doi.org\/10.1145\/3652951","relation":{},"ISSN":["2691-199X","2639-0175"],"issn-type":[{"value":"2691-199X","type":"print"},{"value":"2639-0175","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,12]]},"assertion":[{"value":"2023-12-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-11","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}