{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T00:56:24Z","timestamp":1784336184882,"version":"3.55.0"},"reference-count":101,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T00:00:00Z","timestamp":1724976000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T00:00:00Z","timestamp":1724976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The race to train language models on vast, diverse and inconsistently documented datasets raises pressing legal and ethical concerns. To improve data transparency and understanding, we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace more than 1,800 text datasets. We develop tools and standards to trace the lineage of these datasets, including their source, creators, licences and subsequent use. Our landscape analysis highlights sharp divides in the composition and focus of data licenced for commercial use. Important categories including low-resource languages, creative tasks and new synthetic data all tend to be restrictively licenced. We observe frequent miscategorization of licences on popular dataset hosting sites, with licence omission rates of more than 70% and error rates of more than 50%. This highlights a crisis in misattribution and informed use of popular datasets driving many recent breakthroughs. Our analysis of data sources also explains the application of copyright law and fair use to finetuning data. As a contribution to continuing improvements in dataset transparency and responsible use, we release our audit, with an interactive user interface, the Data Provenance Explorer, to enable practitioners to trace and filter on data provenance for the most popular finetuning data collections: <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/www.dataprovenance.org\">www.dataprovenance.org<\/jats:ext-link>.<\/jats:p>","DOI":"10.1038\/s42256-024-00878-8","type":"journal-article","created":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T10:04:18Z","timestamp":1725012258000},"page":"975-987","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":74,"title":["A large-scale audit of dataset licensing and attribution in AI"],"prefix":"10.1038","volume":"6","author":[{"given":"Shayne","family":"Longpre","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2372-2746","authenticated-orcid":false,"given":"Robert","family":"Mahari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naana","family":"Obeng-Marnu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Damien","family":"Sileo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1435-8535","authenticated-orcid":false,"given":"William","family":"Brannon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niklas","family":"Muennighoff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan","family":"Khazam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jad","family":"Kabbara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kartik","family":"Perisetla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyi","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enrico","family":"Shippole","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kurt","family":"Bollacker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tongshuang","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luis","family":"Villa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8053-9983","authenticated-orcid":false,"given":"Sandy","family":"Pentland","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara","family":"Hooker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,30]]},"reference":[{"key":"878_CR1","unstructured":"Chung, H.W. et al. Scaling instruction-finetuned language models. J. Mach. Learn. Res. 25, 1\u221253 (2024)."},{"key":"878_CR2","unstructured":"Taori, R. et al. Stanford alpaca: an instruction-following Llama model. GitHub https:\/\/crfm.stanford.edu\/2023\/03\/13\/alpaca.html (2023)."},{"key":"878_CR3","unstructured":"Geng, X. et al. Koala: a dialogue model for academic research. Berkeley Artificial Intelligence Research https:\/\/bair.berkeley.edu\/blog\/2023\/04\/03\/koala\/ (2023)."},{"key":"878_CR4","unstructured":"Touvron, H. et al. Llama: open and efficient foundation language models. Preprint at https:\/\/arxiv.org\/abs\/2302.13971 (2023)."},{"key":"878_CR5","doi-asserted-by":"crossref","unstructured":"Wang, Y. et al. Self-instruct: aligning language model with self generated instructions. In Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Rogers, A. et al.) 13484\u201313508 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"878_CR6","unstructured":"Anil, R. et al. Palm 2 technical report. Preprint at https:\/\/arxiv.org\/abs\/2305.10403 (2023)."},{"key":"878_CR7","unstructured":"Achiam, J. et al. GPT-4 technical report. Preprint at https:\/\/arxiv.org\/abs\/2303.08774 (2023)."},{"key":"878_CR8","unstructured":"Model card and evaluations for Claude models. Anthropic https:\/\/www-cdn.anthropic.com\/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226\/Model-Card-Claude-2.pdf (Anthropic, 2023)."},{"key":"878_CR9","unstructured":"Yoo, J., Perlin, K., Kamalakara, S. R. & Ara\u00fajo J. G. Scalable training of language models using JAX-pjit and TPUv4. Preprint at https:\/\/arxiv.org\/abs\/2204.06514 (2022)."},{"key":"878_CR10","unstructured":"Wei, J. et al. Finetuned language models are zero-shot learners. In Proc. 2022 International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=gEZrGCozdqR (ICLR, 2022)."},{"key":"878_CR11","unstructured":"Sanh, V. et al. Multitask prompted training enables zero-shot task generalization. In Proc. 2022 International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=9Vrb9D0WI4 (ICLR, 2022)."},{"key":"878_CR12","doi-asserted-by":"crossref","unstructured":"Muennighoff, N. et al. Crosslingual generalization through multitask finetuning. In Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Rogers, A. et al.) 15991\u201316111 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.acl-long.891"},{"key":"878_CR13","unstructured":"Lhoest, Q. et al. Datasets: a community library for natural language processing. In Proc. 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (eds Adel, H. & Shi, S.)175\u2013184 (Association for Computational Linguistics, 2021)."},{"key":"878_CR14","unstructured":"Gao, L. et al. The pile: an 800\u2009GB dataset of diverse text for language modeling. Preprint at https:\/\/arxiv.org\/abs\/2101.00027 (2020)."},{"key":"878_CR15","unstructured":"Penedo, G. et al. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. In Proc. of the 37th International Conference on Neural Information Processing Systems 79155\u201379172 (Curran, 2023)"},{"key":"878_CR16","unstructured":"Wang, Y. et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. In Proc. of the 2022 Conference on Empirical Methods in Natural Language Processing (eds Goldberg, Y. et al.) 5085\u20135109 (Association for Computational Linguistics, 2022)."},{"key":"878_CR17","unstructured":"Longpre, S. et al. The flan collection: designing data and methods for effective instruction tuning. In Proc. of the 40th International Conference on Machine Learning https:\/\/openreview.net\/pdf?id=ZX4uS605XV (2023)."},{"key":"878_CR18","unstructured":"Gaia search tool https:\/\/huggingface.co\/spaces\/spacerini\/gaia (Spacerini, 2021)."},{"key":"878_CR19","unstructured":"Biderman, S., Bicheno, K. & Gao, L. Datasheet for the pile. Preprint at https:\/\/arxiv.org\/abs\/2201.07311 (2022)."},{"key":"878_CR20","doi-asserted-by":"crossref","unstructured":"Dodge, J. et al. Documenting large webtext corpora: a case study on the colossal clean crawled corpus. In Proc. 2021 Conference on Empirical Methods in Natural Language Processing (eds Adel, H. & Shi, S.) 1286\u20131305 (Association for Computational Linguistics, 2021).","DOI":"10.18653\/v1\/2021.emnlp-main.98"},{"key":"878_CR21","unstructured":"Bandy, J. & Vincent, N. Addressing \u2018documentation debt\u2019 in machine learning research: a retrospective datasheet for bookcorpus. In Proc. of the Neural Information Processing Systems Track on Datasets and Benchmarks (eds Vanschoren, J. & Yeung. S.) https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/file\/54229abfcfa5649e7003b83dd4755294-Paper-round1.pdf (2021)."},{"key":"878_CR22","unstructured":"Bommasani, R. et al. The foundation model transparency index. Preprint at https:\/\/arxiv.org\/abs\/2310.12941 (2023)."},{"key":"878_CR23","unstructured":"Tremblay v. OpenAI, Inc., 3:23-cv-03223-AMO (N.D. Cal. 2024)."},{"key":"878_CR24","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/3458723","volume":"64","author":"T Gebru","year":"2021","unstructured":"Gebru, T. et al. Datasheets for datasets. Commun. ACM 64, 86\u201392 (2021).","journal-title":"Commun. ACM"},{"key":"878_CR25","unstructured":"Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https:\/\/arxiv.org\/abs\/2307.09288 (2023)."},{"key":"878_CR26","doi-asserted-by":"publisher","unstructured":"Sambasivan, N. et al. \u2018Everyone wants to do the model work, not the data work\u2019: data cascades in high-stakes AI. In Proc. 2021 CHI Conference on Human Factors in Computing Systems (eds Kitamura, Y. & Quigley, A.) https:\/\/doi.org\/10.1145\/3411764.34455 (ACM, 2021).","DOI":"10.1145\/3411764.34455"},{"key":"878_CR27","doi-asserted-by":"crossref","unstructured":"Longpre, S. et al. A pretrainer\u2019s guide to training data: measuring the effects of data age, domain coverage, quality, & toxicity. In Proc. of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (eds Duh, K. et al.) 3245\u20133276 (Association for Computational Linguistics, 2024).","DOI":"10.18653\/v1\/2024.naacl-long.179"},{"key":"878_CR28","doi-asserted-by":"crossref","unstructured":"Elangovan, A., He, J. & Verspoor, K. Memorization vs. generalization: quantifying data leakage in NLP performance evaluation. In Proc. 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (eds Merlo, P. et al.) 1325\u20131335 (ACM, 2021).","DOI":"10.18653\/v1\/2021.eacl-main.113"},{"key":"878_CR29","unstructured":"Carlini, N. et al. Quantifying memorization across neural language models. In Proc. 2023 International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=TatRHT_1cK (ICLR, 2023)."},{"key":"878_CR30","unstructured":"Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with gpt-4. Preprint at https:\/\/arxiv.org\/abs\/2303.12712 (2023)."},{"key":"878_CR31","doi-asserted-by":"crossref","unstructured":"Welbl, J. et al. Challenges in detoxifying language models. In Proc. Findings of the Association for Computational Linguistics: EMNLP 2021 (eds Moens, M.-F. et al.) 2447\u20132469 (ACM, 2021).","DOI":"10.18653\/v1\/2021.findings-emnlp.210"},{"key":"878_CR32","doi-asserted-by":"crossref","unstructured":"Xu, A. et al. Detoxifying language models risks marginalizing minority voices. In Proc. 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Toutanova, K. et al.) 2390\u20132397 (ACM, 2021).","DOI":"10.18653\/v1\/2021.naacl-main.190"},{"key":"878_CR33","doi-asserted-by":"crossref","unstructured":"Pozzobon, L., Ermis, B., Lewis, P. & Hooker, S. On the challenges of using black-box APIs for toxicity evaluation in research. In Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 7595\u20137609 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.emnlp-main.472"},{"key":"878_CR34","unstructured":"Luo, Z. et al. Wizardcoder: empowering code large language models with evol-instruct. In Proc. 12th International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=UnUwSIgK5W (ICLR, 2024)."},{"key":"878_CR35","unstructured":"Frankle, J. Tweet by mosaic ML. Twitter https:\/\/twitter.com\/jefrankle\/status\/1654848529834078208 (2023)."},{"key":"878_CR36","unstructured":"Andersen v. Stability AI Ltd., 23-cv-00201-WHO (N.D. Cal. 2023)."},{"key":"878_CR37","unstructured":"Cen, S. H. et al. AI supply chains (and why they matter). The second post in our series On AI Deployment. Substack https:\/\/aipolicy.substack.com\/p\/supply-chains-2 (2023)."},{"key":"878_CR38","doi-asserted-by":"crossref","unstructured":"Bommasani, R., Soylu, D., Liao, T. I., Creel, K. A. & Liang, P. Ecosystem graphs: the social footprint of foundation models. Preprint at https:\/\/arxiv.org\/abs\/2303.15772 (2023).","DOI":"10.21203\/rs.3.rs-2961271\/v1"},{"key":"878_CR39","unstructured":"Ouyang, L. et al. Training language models to follow instructions with human feedback. In Proc. of the 36th International Conference on Neural Information Processing Systems 27730\u201327744 (Curran, 2024)."},{"key":"878_CR40","doi-asserted-by":"crossref","unstructured":"Mitchell, M. et al. Model cards for model reporting. In Proc. Conference on Fairness, Accountability, and Transparency 220\u2013229 (ACM, 2019).","DOI":"10.1145\/3287560.3287596"},{"key":"878_CR41","unstructured":"Wang, Y. et al. Super-natural instructions: generalization via declarative instructions on 1600+ NLP tasks. In Proc. 2022 Conference on Empirical Methods in Natural Language Processing (eds Goldberg, Y. et a.) 5085\u20135109 (Association for Computational Linguistics 2022)."},{"key":"878_CR42","unstructured":"Xu, C. et al. WizardLM: empowering large language models to follow complex instructions. In Proc. 12th International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=CfXh93NDgH (ICLR,2024)."},{"key":"878_CR43","doi-asserted-by":"crossref","unstructured":"Talat, Z. et al. You reap what you sow: on the challenges of bias evaluation under multilingual settings. In Proc. BigScience Episode #5\u2013Workshop on Challenges & Perspectives in Creating Large Language Models (eds Fan, A. et al.) 26\u201341 (Association for Computational Linguistics, 2022).","DOI":"10.18653\/v1\/2022.bigscience-1.3"},{"key":"878_CR44","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1162\/tacl_a_00447","volume":"10","author":"J Kreutzer","year":"2022","unstructured":"Kreutzer, J. et al. Quality at a glance: an audit of web-crawled multilingual datasets. Trans. Assoc. Comput. Linguistics 10, 50\u201372 (2022).","journal-title":"Trans. Assoc. Comput. Linguistics"},{"key":"878_CR45","unstructured":"Shankar, S. et al. No classification without representation: assessing geodiversity issues in open data sets for the developing world. Preprint at https:\/\/arxiv.org\/abs\/1711.08536 (2017)."},{"key":"878_CR46","unstructured":"De Vries, T., Misra, I., Wang, C. & Van der Maaten, L. Does object recognition work for everyone? In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops 52\u201359 (IEEE, 2019)."},{"key":"878_CR47","unstructured":"Mahadev, R. & Chakravarti, A. Understanding gender and racial disparities in image recognition models. Preprint at https:\/\/arxiv.org\/abs\/2107.09211 (2021)."},{"key":"878_CR48","doi-asserted-by":"crossref","unstructured":"Ahia, O., Kreutzer, J. & Hooker, S. The low-resource double bind: an empirical study of pruning for low-resource machine translation. In Proc. Findings of the Association for Computational Linguistics: EMNLP 2021 (eds Moens, F.-M. et al.) 3316\u20133333 (ACM, 2021).","DOI":"10.18653\/v1\/2021.findings-emnlp.282"},{"key":"878_CR49","doi-asserted-by":"publisher","first-page":"1110","DOI":"10.1126\/science.adh4451","volume":"380","author":"Z Epstein","year":"2023","unstructured":"Epstein, Z. et al. Art and the science of generative AI. Science 380, 1110\u20131111 (2023).","journal-title":"Science"},{"key":"878_CR50","first-page":"1407","volume":"36","author":"J Quang","year":"2021","unstructured":"Quang, J. Does training AI violate copyright law? Berkeley Technol. L. J. 36, 1407 (2021).","journal-title":"Berkeley Technol. L. J."},{"key":"878_CR51","unstructured":"Lee, K., Cooper, A. F. & Grimmelmann, J. Talkin \u2019bout AI generation: copyright and the generative-AI supply chain. J. Copyright Soc. USA (in the press)."},{"key":"878_CR52","first-page":"1111","volume":"52","author":"DJ Gervais","year":"2021","unstructured":"Gervais, D. J. AI derivatives: the application to the derivative work right to literary and artistic productions of AI machines. Seton Hall Law Rev. 52, 1111 (2021).","journal-title":"Seton Hall Law Rev."},{"key":"878_CR53","doi-asserted-by":"crossref","unstructured":"Henderson, P. et al. Foundation models and fair use. J. Mach. Learn. Res. 24, 1\u201379 (2023).","DOI":"10.2139\/ssrn.4404340"},{"key":"878_CR54","first-page":"743","volume":"99","author":"MA Lemley","year":"2020","unstructured":"Lemley, M. A. & Casey, B. Fair learning. Texas L. Rev. 99, 743 (2020).","journal-title":"Texas L. Rev."},{"key":"878_CR55","first-page":"45","volume":"41","author":"BLW Sobel","year":"2017","unstructured":"Sobel, B. L. W. Artificial intelligence\u2019s fair use crisis. Columbia J. L. Arts 41, 45\u201397 (2017).","journal-title":"Columbia J. L. Arts"},{"key":"878_CR56","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1126\/science.adi0656","volume":"381","author":"P Samuelson","year":"2023","unstructured":"Samuelson, P. Generative AI meets copyright. Science 381, 158\u2013161 (2023).","journal-title":"Science"},{"key":"878_CR57","unstructured":"Doe v. GitHub, Inc., 22-cv-06823-JST (N.D. Cal. 2024)."},{"key":"878_CR58","unstructured":"Bill Graham Archives v. Dorling Kindersley Ltd., 448 F.3d 605 (2d Cir. 2006)."},{"key":"878_CR59","first-page":"141","volume":"53","author":"CA Grossman","year":"2005","unstructured":"Grossman, C. A. From Sony to Grokster, the failure of the copyright doctrines of contributory infringement and vicarious liability to resolve the war between content and destructive technologies. Buffalo L. Rev. 53, 141\u2013268 (2005).","journal-title":"Buffalo L. Rev."},{"key":"878_CR60","unstructured":"Marks, C. P. & Moll, D. K. The Law of Business Torts and Unfair Competition: Cases, Materials, and Problems. American Casebook Series (West Academic, 2023)."},{"key":"878_CR61","unstructured":"Victor, J. & Efrati, A. Alphabet\u2019s Google and DeepMind pause grudges, join forces to chase OpenAI. The Information https:\/\/www.theinformation.com\/articles\/alphabets-google-and-deepmind-pause-grudges-join-forces-to-chase-openai (2023)."},{"key":"878_CR62","unstructured":"Suggs, N. & Venables, P. Protecting customers with generative AI indemnification. Google Cloud https:\/\/cloud.google.com\/blog\/products\/ai-machine-learning\/protecting-customers-with-generative-ai-indemnification (2023)."},{"key":"878_CR63","unstructured":"Mahari, R. et al. Comment to U.S. copyright office on data provenance and copyright (US Copyright Office, 2023); https:\/\/dspace.mit.edu\/handle\/1721.1\/154171"},{"key":"878_CR64","doi-asserted-by":"publisher","unstructured":"Longpre, S. et al. Position: data authenticity, consent, & provenance for AI are all broken: what will it take to fix them? An MIT Exploration of Generative AI https:\/\/doi.org\/10.21428\/e4baedd9.a650f77d (2024).","DOI":"10.21428\/e4baedd9.a650f77"},{"key":"878_CR65","unstructured":"Kinney, R. M. et al. The semantic scholar open data platform. Preprint at https:\/\/arxiv.org\/abs\/2301.10140 (2023)."},{"key":"878_CR66","unstructured":"Petrov, A., La Malfa, E., Torr, P. & Bibi, A. Language model tokenizers introduce unfairness between languages. In Proc. of the 37th International Conference on Neural Information Processing Systems 36963\u201336990 (Curran, 2024)."},{"key":"878_CR67","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1162\/tacl_a_00041","volume":"6","author":"EM Bender","year":"2018","unstructured":"Bender, E. M. & Friedman, B. Data statements for natural language processing: toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguistics 6, 587\u2013604 (2018).","journal-title":"Trans. Assoc. Comput. Linguistics"},{"key":"878_CR68","doi-asserted-by":"crossref","unstructured":"Pushkarna, M., Zaldivar, A. & Kjartansson, O. Data cards: purposeful and transparent dataset documentation for responsible AI. In Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 1776\u20131826 (ACM, 2022).","DOI":"10.1145\/3531146.3533231"},{"key":"878_CR69","doi-asserted-by":"publisher","unstructured":"Bender, E. M. On achieving and evaluating language-independence in NLP. Linguist. Issues Lang. Technol. https:\/\/doi.org\/10.33011\/lilt.v6i.1239 (2011).","DOI":"10.33011\/lilt.v6i.1239"},{"key":"878_CR70","doi-asserted-by":"publisher","unstructured":"Longpre, S. et al. Data-Provenance-Initiative\/Data-Provenance-Collection: Data Provenance Initiative Release. Zenodo https:\/\/doi.org\/10.5281\/zenodo.11587503 (2024).","DOI":"10.5281\/zenodo.11587503"},{"key":"878_CR71","unstructured":"Durbin, J. Airoboros: using large language models to fine-tune large language models. GitHub https:\/\/github.com\/jondurbin\/airoboros (2023)."},{"key":"878_CR72","unstructured":"Bai, Y. et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. Preprint at https:\/\/arxiv.org\/abs\/2204.05862 (2022)."},{"key":"878_CR73","unstructured":"Ganguli, D. et al. Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. Preprint at https:\/\/arxiv.org\/abs\/2209.07858 (2022)."},{"key":"878_CR74","doi-asserted-by":"crossref","unstructured":"Xu, C., Guo, D., Duan, N. & McAuley, J. Baize: an open-source chat model with parameter-efficient tuning on self-chat data. In Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 6268\u20136278 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.emnlp-main.385"},{"key":"878_CR75","doi-asserted-by":"crossref","unstructured":"Kry\u015bci\u0144ski, W., Rajani, N., Agarwal, D., Xiong, C. & Radev, D. Booksum: a collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022 (eds Goldberg, Y. et al.) 6536\u20136558 (Association for Computational Linguistics, 2022).","DOI":"10.18653\/v1\/2022.findings-emnlp.488"},{"key":"878_CR76","unstructured":"Li, G., Hammoud, H., Itani, H., Khizbullin, D. & Ghanem, B. CAMEL: communicative agents for \u2018mind\u2019 exploration of large scale language model society. In Proc. of the 37th International Conference on Neural Information Processing Systems 51991\u201352008 (Curran, 2024)."},{"key":"878_CR77","doi-asserted-by":"crossref","unstructured":"Kim, S. et al. The CoT collection: improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 12685\u201312708 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.emnlp-main.782"},{"key":"878_CR78","unstructured":"Muennighoff, N. et al. Octopack: instruction tuning code large language models. In Proc. 12th International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=mw1PWNSWZP (ICLR, 2024)."},{"key":"878_CR79","unstructured":"Conover, M. et al. Free Dolly: introducing the world\u2019s first truly open instruction-tuned LLM. Databricks www.databricks.com\/blog\/2023\/04\/12\/dolly-first-open-commercially-viable-instruction-tuned-llm (2023)."},{"key":"878_CR80","unstructured":"Peng, B., Li, C., He, P., Galley, M. & Gao, J. Instruction tuning with GPT-4. Preprint at https:\/\/arxiv.org\/abs\/2304.03277 (2023)."},{"key":"878_CR81","unstructured":"Anand, Y., Nussbaum, Z., Duderstadt, B., Schmidt, B. & Mulyar, A. GPT4all: training an assistant-style chatbot with large scale data distillation from GPT-3.5-turbo. GitHub https:\/\/github.com\/nomic-ai\/gpt4all (2023)."},{"key":"878_CR82","unstructured":"Patil, S. G., Zhang, T., Wang, X. & Gonzalez, J. E. Gorilla: large language model connected with massive APIs. Preprint at arXiv https:\/\/arxiv.org\/abs\/2305.15334 (2023)."},{"key":"878_CR83","unstructured":"Guo, B. et al. How close is ChatGPT to human experts? comparison corpus, evaluation, and detection. Preprint at https:\/\/arxiv.org\/abs\/2301.07597 (2023)."},{"key":"878_CR84","unstructured":"Nguyen, H., Suri, S., Tsui, K. & Schuhmann, C. The Open Instruction Generalist (OIG) Dataset (LAION, 2023); https:\/\/laion.ai\/blog\/oig-dataset\/"},{"key":"878_CR85","unstructured":"Zhou, C. et al. Lima: Less is more for alignment. In Proc. of the 37th International Conference on Neural Information Processing Systems 55006\u201355021 (Curran, 2024)."},{"key":"878_CR86","unstructured":"K\u00f6ksal, A., Schick, T., Korhonen, A. & Sch\u00fctze, H. Longform: optimizing instruction tuning for long text generation with corpus extraction. Preprint at https:\/\/arxiv.org\/abs\/2304.08460 (2023)."},{"key":"878_CR87","unstructured":"Stiennon, N. et al. Learning to summarize from human feedback. In Proc. of the 34th International Conference on Neural Information Processing Systems 3008\u20133021 (Curran, 2020)."},{"key":"878_CR88","unstructured":"K\u00f6pf, A. et al. OpenAssistant conversations\u2013democratizing large language model alignment. In Proc. of the 37th International Conference on Neural Information Processing Systems 47669\u201347681 (Curran, 2024)."},{"key":"878_CR89","unstructured":"Mukherjee, S. et al. Orca: progressive learning from complex explanation traces of GPT-4. Preprint at https:\/\/arxiv.org\/abs\/2306.02707 (2023)."},{"key":"878_CR90","unstructured":"Ethayarajh, K., Zhang, H., Wang, Y. & Jurafsky, D. Stanford Guman Preferences Dataset (2023); https:\/\/huggingface.co\/datasets\/stanfordnlp\/SHP"},{"key":"878_CR91","unstructured":"Vercel. Sharegpt https:\/\/sharegpt.com\/ (2023)."},{"key":"878_CR92","unstructured":"Li, R. et al. Starcoder: may the source be with you! Trans. Mach. Learn. Res. https:\/\/openreview.net\/pdf?id=KoFOg41haE (2023)."},{"key":"878_CR93","unstructured":"Sileo, D. tasksource: a dataset harmonization framework for streamlined NLP multi-task learning and evaluation. Preprint at https:\/\/arxiv.org\/abs\/2301.05948 (2023)."},{"key":"878_CR94","unstructured":"Weston, J. et al. Towards AI-complete question answering: a set of prerequisite toy tasks. In Proc. of the 4th International Conference on Learning Representations (eds Bengio, Y. & and LeCun, Y.) (ICLR, 2016)."},{"key":"878_CR95","unstructured":"Eldan, R. & Li, Y. Tinystories: how small can language models be and still speak coherent english? Preprint at https:\/\/arxiv.org\/abs\/2305.07759 (2023)."},{"key":"878_CR96","unstructured":"Qin, Y. et al. ToolLLM: facilitating large language models to master 16000+ real-world APIs. In Proc. 2024 International Conference on Learning Representations https:\/\/openreview.net\/pdf?id=dHng2O0Jjr (ICLR, 2024)."},{"key":"878_CR97","doi-asserted-by":"crossref","unstructured":"Ding, N. et al. Enhancing chat language models by scaling high-quality instructional conversations. In Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 3029\u20133051 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.emnlp-main.183"},{"key":"878_CR98","doi-asserted-by":"crossref","unstructured":"Honovich, O., Scialom, T., Levy, O. & Schick, T. Unnatural instructions: tuning language models with (almost) no human labor. In Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds Rogers, A. et al.) 14409\u201314428 (Association for Computational Linguistics, 2023).","DOI":"10.18653\/v1\/2023.acl-long.806"},{"key":"878_CR99","unstructured":"Nakano, R. et al. WebGPT: browser-assisted question-answering with human feedback. Preprint at https:\/\/arxiv.org\/abs\/2112.09332 (2021)."},{"key":"878_CR100","unstructured":"Hendrycks, D. et al. Measuring massive multitask language understanding. In Proc. International Conference on Learning Representations (2020)."},{"key":"878_CR101","unstructured":"Srivastava, A. et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. https:\/\/openreview.net\/pdf?id=uyTL5Bvosj (2023)."}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00878-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00878-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00878-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T10:08:24Z","timestamp":1725012504000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00878-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,30]]},"references-count":101,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["878"],"URL":"https:\/\/doi.org\/10.1038\/s42256-024-00878-8","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,30]]},"assertion":[{"value":"15 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 August 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The following authors are employed by a firm engaged in AI or related research: N.M. is a Research Engineer at Contextual AI. K.P. is a Research Scientist at Apple. E.S. is CEO of Teraflop AI. K.B. is Director of Engineering at MLCommons. L.V. is cofounder and general counsel of Tidelift. S.H. is head of Cohere For AI. The other authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}