{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T18:32:20Z","timestamp":1786991540282,"version":"build-2736575974"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T00:00:00Z","timestamp":1782864000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/publication-rights-and-licensing-policy"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGCOMM Comput. Commun. Rev."],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:p>We present a large-scale measurement of emerging Artificial Intelligence (AI)-oriented web permission and descriptor mechanisms across approximately 4M domains. Beyond traditional robots.txt files, we analyze the adoption, structure, and interactions of newer files, including the permission-oriented ai.txt and the descriptor-oriented llms.txt and llms-full.txt. While robots.txt remains the dominant mechanism for signaling crawler permissions, adoption of AI-specific permission files is emerging, especially among technology-focused domains. Directive usage shows a strong emphasis on restricting AI agents, with prevalent disallow rules and relatively few AI-specific permission controls. Crossfile comparisons reveal that conflicts between robots.txt and ai.txt are infrequent, while some llms.txt link targets point to content blocked by access-control files.<\/jats:p>","DOI":"10.1145\/3831956.3831960","type":"journal-article","created":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T18:20:36Z","timestamp":1786990836000},"page":"29-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI"],"prefix":"10.1145","volume":"56","author":[{"given":"Felix","family":"Hoffmann","sequence":"first","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tobias","family":"Goergens","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sonal","family":"Khosla","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vaibhav","family":"Bajpai","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671900"},{"key":"e_1_2_1_2_1","unstructured":"Will Allen. 2025. Giving Users Choice with Cloudflare's New Content Signals Policy. The Cloudflare Blog. https:\/\/blog.cloudflare.com\/content-signals-policy\/"},{"key":"e_1_2_1_3_1","unstructured":"Fabrice Canel and Krishna Madhavan. 2025. Robots Exclusion Protocol Extension to Communicate AI Preferences Vocabulary. Internet-Draft draft-canel-robots-ai-control-01. Internet Engineering Task Force. https:\/\/datatracker.ietf.org\/doc\/draft-canel-robots-ai-control\/01\/"},{"key":"e_1_2_1_4_1","unstructured":"Cloudflare. 2025. Content Signals. https:\/\/contentsignals.org\/"},{"key":"e_1_2_1_5_1","unstructured":"Cloudflare. 2025. Get Multiple Domain Details. Cloudflare API. https:\/\/developers.cloudflare.com\/api\/resources\/intel\/subresources\/domains\/subresources\/bulks\/methods\/get\/"},{"key":"e_1_2_1_6_1","unstructured":"DeviantArt. 2022. UPDATE All Deviations Are Opted Out of AI Datasets. DeviantArt. https:\/\/www.deviantart.com\/team\/journal\/UPDATE-All-Deviations-Are-Opted-Out-of-AI-Datasets-934500371"},{"key":"e_1_2_1_7_1","unstructured":"DistriNet. 2025. Tranco. DistriNet. https:\/\/tranco-list.eu\/"},{"key":"e_1_2_1_8_1","unstructured":"European Parliament and Council of the European Union. 2024. Regulation (EU) 2024\/1689 (Artificial Intelligence Act). Official Journal of the European Union. https:\/\/eur-lex.europa.eu\/eli\/reg\/2024\/1689\/oj"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-026-09779-8"},{"key":"e_1_2_1_10_1","volume-title":"Meet Llms.Txt, a Proposed Standard for AI Website Content Crawling. https:\/\/searchengineland.com\/llms-txt-proposed-standard-453676","author":"Garner Rob","year":"2025","unstructured":"Rob Garner. 2025. Meet Llms.Txt, a Proposed Standard for AI Website Content Crawling. https:\/\/searchengineland.com\/llms-txt-proposed-standard-453676"},{"key":"e_1_2_1_11_1","unstructured":"Google. 2025. Robots Meta Tags Specifications. Google for Developers. https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots-meta-tag Accessed: 2026-06-24."},{"key":"e_1_2_1_12_1","unstructured":"Jeremy Howard. 2024. The \/llms.txt File. llms-txt. https:\/\/llmstxt.org\/ Accessed: 2025-11-20."},{"key":"e_1_2_1_13_1","unstructured":"Gary Illyes and Martin Thomson. 2025. Associating AI Usage Preferences with Content in HTTP. Internet-Draft draft-ietf-aipref-attach-04. Internet Engineering Task Force. https:\/\/datatracker.ietf.org\/doc\/draft-ietf-aipref-attach\/04\/"},{"key":"e_1_2_1_14_1","unstructured":"Paul Keller and Martin Thomson. 2025. A Vocabulary for Expressing AI Usage Preferences. Internet-Draft draft-ietf-aipref-vocab-04. Internet Engineering Task Force. https:\/\/datatracker.ietf.org\/doc\/draft-ietf-aipref-vocab\/04\/"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3730567.3764471"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Martijn Koster Gary Illyes Henner Zeller and Lizzi Sassman. 2022. Robots Exclusion Protocol. RFC 9309. 10.17487\/RFC9309","DOI":"10.17487\/RFC9309"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","unstructured":"Yuekang Li Wei Song Bangshuo Zhu Dong Gong Yi Liu Gelei Deng Chunyang Chen Lei Ma Jun Sun Toby Walsh and Jingling Xue. 2025. Ai.Txt: A Domain-Specific Language for Guiding AI Interactions with the Internet. 10.48550\/ARXIV.2505.07834","DOI":"10.48550\/ARXIV.2505.07834"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3730567.3732913"},{"key":"e_1_2_1_19_1","unstructured":"llms.txt hub team. 2025. llms.txt hub. https:\/\/llmstxthub.com\/"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-3431"},{"key":"e_1_2_1_21_1","unstructured":"Majestic. 2025. OpenRobotsTXT. https:\/\/openrobotstxt.org\/"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","unstructured":"Samuele Marro and Philip Torr. 2025. LLM Agents Are the Antidote to Walled Gardens. arXiv:2506.23978 [cs] 10.48550\/arXiv.2506.23978","DOI":"10.48550\/arXiv.2506.23978"},{"key":"e_1_2_1_23_1","unstructured":"Mozilla. 2025. <meta Name=\"robots\"> - HTML. MDN Web Docs. https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTML\/Reference\/Elements\/meta\/name\/robots"},{"key":"e_1_2_1_24_1","unstructured":"Mozilla. 2025. X-Robots-Tag Header - HTTP. MDN Web Docs. https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTTP\/Reference\/Headers\/X-Robots-Tag"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1609\/aies.v8i2.36685"},{"key":"e_1_2_1_26_1","unstructured":"Natallie Rocha. 2025. Cloudflare Introduces Default Blocking of A.I. Data Scrapers. The New York Times. https:\/\/www.nytimes.com\/2025\/07\/01\/technology\/cloudflare-ai-data.html"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","unstructured":"Jordi Montes Sanabria and Pol Alvarez Vecino. 2025. Beyond the Sum: Unlocking AI Agents Potential Through Market Forces. arXiv:2501.10388 [cs] 10.48550\/arXiv.2501.10388","DOI":"10.48550\/arXiv.2501.10388"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242726"},{"key":"e_1_2_1_29_1","unstructured":"Michael Tremante and Leah Romm. 2025. Vocabulary for Expressing Content Signals. Internet-Draft draft-romm-aipref-contentsignals-00. Internet Engineering Task Force. https:\/\/datatracker.ietf.org\/doc\/draft-romm-aipref-contentsignals\/00\/"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","unstructured":"Avinash Tripathi. 2025. A Strategic Outlook on LLM SEO: Using FileFormat Logic to Guide AI-Optimized Content Design. SSRN Working Paper. 10.2139\/ssrn.5375528","DOI":"10.2139\/ssrn.5375528"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.09798"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","unstructured":"Yoel Zimmermann Adib Bazgir Zartashia Afzal et al. 2025. Reflections from the 2024 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry. arXiv:2411.15221 [cs] 10.48550\/arXiv.2411.15221","DOI":"10.48550\/arXiv.2411.15221"}],"container-title":["ACM SIGCOMM Computer Communication Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3831956.3831960","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T18:21:02Z","timestamp":1786990862000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3831956.3831960"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,1]]},"references-count":32,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,7,1]]}},"alternative-id":["10.1145\/3831956.3831960"],"URL":"https:\/\/doi.org\/10.1145\/3831956.3831960","relation":{},"ISSN":["0146-4833"],"issn-type":[{"value":"0146-4833","type":"print"}],"subject":[],"published":{"date-parts":[[2026,7,1]]},"assertion":[{"value":"2026-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}