{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T12:55:00Z","timestamp":1776171300785,"version":"3.50.1"},"reference-count":39,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T00:00:00Z","timestamp":1706832000000},"content-version":"vor","delay-in-days":32,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Partnership for the Organization of Innovation and New Technologies"},{"DOI":"10.13039\/501100000155","name":"Social Sciences and Humanities Research Council of Canada","doi-asserted-by":"publisher","award":["895-2018-1006"],"award-info":[{"award-number":["895-2018-1006"]}],"id":[{"id":"10.13039\/501100000155","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/W013770\/1"],"award-info":[{"award-number":["BB\/W013770\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018569","name":"Snap Research Fellowship","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100018569","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We put forward a novel approach using a generative language model (GPT-4) to produce labels and rationales for large-scale text analysis. The approach is used to discover public value expressions in patents. Using text (5.4 million sentences) for 154,934 US AI patent documents from the United States Patent and Trademark Office (USPTO), we design a semi-automated, human-supervised framework for identifying and labeling public value expressions in these sentences. A GPT-4 prompt is developed that includes definitions, guidelines, examples, and rationales for text classification. We evaluate the labels and rationales produced by GPT-4 using BLEU scores and topic modeling, finding that they are accurate, diverse, and faithful. GPT-4 achieved an advanced recognition of public value expressions from our framework, which it also uses to discover unseen public value expressions. The GPT-produced labels are used to train BERT-based classifiers and predict sentences on the entire database, achieving high F1 scores for the 3-class (0.85) and 2-class classification (0.91) tasks. We discuss the implications of our approach for conducting large-scale text analyses with complex and abstract concepts. With careful framework design and interactive human oversight, we suggest that generative language models can offer significant assistance in producing labels and rationales.<\/jats:p>","DOI":"10.1162\/qss_a_00285","type":"journal-article","created":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T18:51:15Z","timestamp":1706899875000},"page":"153-169","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":27,"title":["Large-scale text analysis using generative language models: A case study in discovering public value expressions in AI patents"],"prefix":"10.1162","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2640-5623","authenticated-orcid":true,"given":"Sergio","family":"Pelaez","sequence":"first","affiliation":[{"name":"School of Public Policy, Georgia Institute of Technology, Atlanta, GA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6182-9857","authenticated-orcid":true,"given":"Gaurav","family":"Verma","sequence":"additional","affiliation":[{"name":"School of Computational Science and Engineering, College of Computing, Georgia Institute of Technology, Atlanta, GA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5230-1695","authenticated-orcid":true,"given":"Barbara","family":"Ribeiro","sequence":"additional","affiliation":[{"name":"SKEMA Business School, Universit\u00e9 C\u00f4te d\u2019Azur, Campus Grand Paris, Paris, France"},{"name":"Manchester Institute of Innovation Research, University of Manchester, Manchester, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2488-5985","authenticated-orcid":true,"given":"Philip","family":"Shapira","sequence":"additional","affiliation":[{"name":"School of Public Policy, Georgia Institute of Technology, Atlanta, GA, USA"},{"name":"Manchester Institute of Innovation Research, University of Manchester, Manchester, UK"}]}],"member":"281","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"issue":"3","key":"2024052115144499500_bib1","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1111\/radm.12408","article-title":"The application of text mining methods in innovation research: Current state, evolution patterns, and development priorities","volume":"50","author":"Antons","year":"2020","journal-title":"R&D Management"},{"issue":"2","key":"2024052115144499500_bib2","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1017\/S0003055416000058","article-title":"Crowd-sourced text analysis: Reproducible and agile production of political data","volume":"110","author":"Benoit","year":"2016","journal-title":"American Political Science Review"},{"key":"2024052115144499500_bib3","first-page":"993","article-title":"Latent Dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"2024052115144499500_bib4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.00612","article-title":"Eight things to know about large language models","author":"Bowman","year":"2023","journal-title":"arXiv"},{"issue":"2","key":"2024052115144499500_bib5","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1111\/0033-3352.00165","article-title":"Public-value failure: When efficient markets may not do","volume":"62","author":"Bozeman","year":"2002","journal-title":"Public Administration Review"},{"issue":"1","key":"2024052115144499500_bib6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11024-011-9161-7","article-title":"Public value mapping and science policy evaluation","volume":"49","author":"Bozeman","year":"2011","journal-title":"Minerva"},{"key":"2024052115144499500_bib8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.12712","article-title":"Sparks of artificial general intelligence: Early experiments with GPT-4","author":"Bubeck","year":"2023","journal-title":"arXiv"},{"key":"2024052115144499500_bib7","doi-asserted-by":"publisher","first-page":"101475","DOI":"10.1016\/j.techsoc.2020.101475","article-title":"Towards a deliberative framework for responsible innovation in artificial intelligence","volume":"64","author":"Buhmann","year":"2021","journal-title":"Technology in Society"},{"key":"2024052115144499500_bib9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.11416","article-title":"Scaling instruction-finetuned language models","author":"Chung","year":"2022","journal-title":"arXiv"},{"key":"2024052115144499500_bib10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv"},{"issue":"1","key":"2024052115144499500_bib11","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1162\/qss_a_00106","article-title":"Fine-grained classification of social science journal articles using textual data: A comparison of supervised machine learning approaches","volume":"2","author":"Eykens","year":"2021","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"2024052115144499500_bib12","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1162\/qss_a_00183","article-title":"The Microsoft Academic Knowledge Graph enhanced: Author name disambiguation, publication classification, and embeddings","volume":"3","author":"F\u00e4rber","year":"2022","journal-title":"Quantitative Science Studies"},{"issue":"6","key":"2024052115144499500_bib13","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1177\/0275074018814244","article-title":"Public values theory: What is missing?","volume":"49","author":"Fukumoto","year":"2019","journal-title":"The American Review of Public Administration"},{"key":"2024052115144499500_bib14","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1145\/3351095.3372862","article-title":"Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from?","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","author":"Geiger","year":"2020"},{"issue":"2","key":"2024052115144499500_bib15","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1007\/s10961-021-09900-2","article-title":"Identifying artificial intelligence (AI) invention: A novel AI patent dataset","volume":"47","author":"Giczy","year":"2022","journal-title":"Journal of Technology Transfer"},{"issue":"7","key":"2024052115144499500_bib16","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1002\/smj.3480","article-title":"Using supervised machine learning to scale human-coded data: A method and dataset in the board leadership context","volume":"44","author":"Harrison","year":"2022","journal-title":"Strategic Management Journal"},{"key":"2024052115144499500_bib17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2006.03654","article-title":"DeBERTa: Decoding-enhanced BERT with disentangled attention","author":"He","year":"2020","journal-title":"arXiv"},{"key":"2024052115144499500_bib18","doi-asserted-by":"publisher","DOI":"10.4135\/9781071878781","volume-title":"Content analysis: An introduction to its methodology","author":"Krippendorff","year":"2019"},{"key":"2024052115144499500_bib19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.11942","article-title":"ALBERT: A lite BERT for self-supervised learning of language representations","author":"Lan","year":"2019","journal-title":"arXiv"},{"key":"2024052115144499500_bib20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.02124","article-title":"PatentBERT: Patent classification with fine-tuning a pre-trained BERT model","author":"Lee","year":"2019","journal-title":"arXiv"},{"key":"2024052115144499500_bib21","doi-asserted-by":"publisher","DOI":"10.1177\/23780231211062345","article-title":"Qualitative coding in the computational era: A hybrid approach to improve reliability and reduce effort for coding ethnographic interviews","volume":"7","author":"Li","year":"2021","journal-title":"Socius"},{"issue":"12","key":"2024052115144499500_bib22","doi-asserted-by":"publisher","first-page":"e0262050","DOI":"10.1371\/journal.pone.0262050","article-title":"Mapping technological innovation dynamics in artificial intelligence domains: Evidence from a global patent analysis","volume":"16","author":"Liu","year":"2021","journal-title":"PLOS ONE"},{"key":"2024052115144499500_bib23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv"},{"key":"2024052115144499500_bib24","doi-asserted-by":"publisher","first-page":"1399","DOI":"10.1609\/aaai.v24i1.7514","article-title":"Diversifying query suggestion results","volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence","author":"Ma","year":"2010"},{"key":"2024052115144499500_bib25","volume-title":"AI risk management framework","author":"NIST","year":"2023"},{"key":"2024052115144499500_bib26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.13375","article-title":"Capabilities of GPT-4 on medical challenge problems","author":"Nori","year":"2023","journal-title":"arXiv"},{"key":"2024052115144499500_bib27","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2024052115144499500_bib28","doi-asserted-by":"publisher","DOI":"10.1002\/0471698466","volume-title":"Tech mining: Exploiting new technologies for competitive advantage","author":"Porter","year":"2004"},{"issue":"1","key":"2024052115144499500_bib30","doi-asserted-by":"publisher","first-page":"104607","DOI":"10.1016\/j.respol.2022.104607","article-title":"The digitalisation paradox of everyday scientific labour: How mundane knowledge work is amplified and diversified in the biosciences","volume":"52","author":"Ribeiro","year":"2023","journal-title":"Research Policy"},{"issue":"1","key":"2024052115144499500_bib29","doi-asserted-by":"publisher","first-page":"103875","DOI":"10.1016\/j.respol.2019.103875","article-title":"Private and public values of innovation: A patent analysis of synthetic biology","volume":"49","author":"Ribeiro","year":"2020","journal-title":"Research Policy"},{"issue":"10","key":"2024052115144499500_bib31","doi-asserted-by":"publisher","first-page":"1827","DOI":"10.1016\/j.respol.2015.06.006","article-title":"What is an emerging technology?","volume":"44","author":"Rotolo","year":"2015","journal-title":"Research Policy"},{"issue":"1","key":"2024052115144499500_bib32","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1177\/0275074014525833","article-title":"As good as it gets? On the meaning of public value in the study of policy and management","volume":"45","author":"Rutgers","year":"2015","journal-title":"American Review of Public Administration"},{"issue":"4","key":"2024052115144499500_bib33","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1162\/qss_a_00223","article-title":"AI for AI: Using AI methods for classifying AI science documents","volume":"3","author":"Sachini","year":"2022","journal-title":"Quantitative Science Studies"},{"key":"2024052115144499500_bib34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.01108","article-title":"DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter","author":"Sanh","year":"2019","journal-title":"arXiv"},{"key":"2024052115144499500_bib35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.13971","article-title":"LLaMa: Open and efficient foundation language models","author":"Touvron","year":"2023","journal-title":"arXiv"},{"key":"2024052115144499500_bib36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.02646","article-title":"Robustness of fusion-based multimodal classifiers to cross-modal content dilutions","author":"Verma","year":"2022","journal-title":"arXiv"},{"key":"2024052115144499500_bib37","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.11903","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei","year":"2022","journal-title":"arXiv"},{"key":"2024052115144499500_bib38","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.01068","article-title":"OPT: Open pre-trained transformer language models","author":"Zhang","year":"2022","journal-title":"arXiv"},{"key":"2024052115144499500_bib39","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1145\/3209978.3210080","article-title":"Texygen: A benchmarking platform for text generation models","volume-title":"SIGIR\u201918: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","author":"Zhu","year":"2018"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/1\/153\/2373904\/qss_a_00285.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/5\/1\/153\/2373904\/qss_a_00285.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T11:15:53Z","timestamp":1716290153000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/5\/1\/153\/119275\/Large-scale-text-analysis-using-generative"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00285","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00285\/v2\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}