{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:59:43Z","timestamp":1750309183970,"version":"3.41.0"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA1","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NSF","award":["2313010"],"award-info":[{"award-number":["2313010"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2024,4,29]]},"abstract":"<jats:p>Finding errors in machine learning applications requires a thorough exploration of their behavior over data. Existing approaches used by practitioners are often ad-hoc and lack the abstractions needed to scale this process. We present TorchQL, a programming framework to evaluate and improve the correctness of machine learning applications. TorchQL allows users to write queries to specify and check integrity constraints over machine learning models and datasets. It seamlessly integrates relational algebra with functional programming to allow for highly expressive queries using only eight intuitive operators. We evaluate TorchQL on diverse use-cases including finding critical temporal inconsistencies in objects detected across video frames in autonomous driving, finding data imputation errors in time-series medical records, finding data labeling errors in real-world images, and evaluating biases and constraining outputs of language models. Our experiments show that TorchQL enables up to 13x faster query executions than baselines like Pandas and MongoDB, and up to 40% shorter queries than native Python. We also conduct a user study and find that TorchQL is natural enough for developers familiar with Python to specify complex integrity constraints.<\/jats:p>","DOI":"10.1145\/3649841","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T17:53:50Z","timestamp":1714413230000},"page":"833-863","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TorchQL: A Programming Framework for Integrity Constraints in Machine Learning"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3100-0455","authenticated-orcid":false,"given":"Aaditya","family":"Naik","sequence":"first","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1887-100X","authenticated-orcid":false,"given":"Adam","family":"Stein","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9770-5765","authenticated-orcid":false,"given":"Yinjun","family":"Wu","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1348-8618","authenticated-orcid":false,"given":"Mayur","family":"Naik","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8568-6659","authenticated-orcid":false,"given":"Eric","family":"Wong","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"2022. Stream Analytics Query Language Reference. https:\/\/learn.microsoft.com\/en-us\/stream-analytics-query\/stream-analytics-query-language-reference"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687625"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462624"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.07.015"},{"key":"e_1_2_2_5_1","unstructured":"ArangoDB. 2023. ArangoDB Query Language (AQL) Introduction: ArangoDB Documentation. www.arangodb.com"},{"key":"e_1_2_2_6_1","unstructured":"Sean Augenstein H Brendan McMahan Daniel Ramage Swaroop Ramaswamy Peter Kairouz Mingqing Chen and Rajiv Mathews. 2019. Generative models for effective ML on private decentralized datasets. arXiv preprint arXiv:1911.06679."},{"volume-title":"MongoDB in action: covers MongoDB version 3.0","author":"Banker Kyle","key":"e_1_2_2_7_1","unstructured":"Kyle Banker, Douglas Garrett, Peter Bakkum, and Shaun Verch. 2016. MongoDB in action: covers MongoDB version 3.0. Simon and Schuster."},{"key":"e_1_2_2_8_1","unstructured":"Sara Beery Elijah Cole and Arvi Gjoka. 2020. The iWildCam 2020 Competition Dataset. arXiv preprint arXiv:2004.10340."},{"key":"e_1_2_2_9_1","volume-title":"Proceedings of the ACM on Programming Languages, 7, PLDI (2023)","author":"Beurer-Kellner Luca","year":"2023","unstructured":"Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7, PLDI (2023), 1946\u20131969."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2014.10.083"},{"key":"e_1_2_2_11_1","volume-title":"Proceedings, Part III 16","author":"Bolya Daniel","year":"2020","unstructured":"Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. 2020. Tide: A general toolbox for identifying object detection errors. In Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK, August 23\u201328, 2020, Proceedings, Part III 16. 558\u2013573."},{"key":"e_1_2_2_12_1","volume-title":"ICML Workshop on Reliable Machine Learning in the Wild. 103","author":"Cadamuro Gabriel","year":"2016","unstructured":"Gabriel Cadamuro, Ran Gilad-Bachrach, and Xiaojin Zhu. 2016. Debugging machine learning models. In ICML Workshop on Reliable Machine Learning in the Wild. 103."},{"key":"e_1_2_2_13_1","unstructured":"Harrison Chase. 2022. LangChain. https:\/\/github.com\/langchain-ai\/langchain"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462180"},{"key":"e_1_2_2_15_1","volume-title":"Now Publishers Inc","author":"Chirkova Rada","year":"1986","unstructured":"Rada Chirkova and Jun Yang. 2012. Materialized Views. Now Publishers Inc., Hanover, MA, USA. isbn:160198622X"},{"key":"e_1_2_2_16_1","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton and Reiichiro Nakano. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/362384.362685"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.350"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.119619"},{"key":"e_1_2_2_21_1","volume-title":"Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, issn:2835-8856 https:\/\/openreview.net\/forum?id=uyTL5Bvosj","author":"Aarohi Srivastava","year":"2023","unstructured":"Aarohi Srivastava et al.. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, issn:2835-8856 https:\/\/openreview.net\/forum?id=uyTL5Bvosj"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314633"},{"key":"e_1_2_2_23_1","volume-title":"Database Systems: The Complete Book (2 ed.)","author":"Garcia-Molina Hector","year":"2008","unstructured":"Hector Garcia-Molina, Jeffrey D. Ullman, and Jennifer Widom. 2008. Database Systems: The Complete Book (2 ed.). Prentice Hall Press, USA. isbn:9780131873254"},{"volume-title":"Logics for databases and information systems","author":"Godfrey Parke","key":"e_1_2_2_24_1","unstructured":"Parke Godfrey, John Grant, Jarek Gryz, and Jack Minker. 1998. Integrity constraints: Semantics and applications. In Logics for databases and information systems. Springer."},{"key":"e_1_2_2_25_1","unstructured":"Priya Goyal Piotr Doll\u00e1r Ross Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia and Kaiming He. 2017. Accurate large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677."},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. 202\u2013214","author":"Havaldar Shreya","year":"2023","unstructured":"Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023. Multilingual Language Models are not Multicultural: A Case Study in Emotion. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. 202\u2013214."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_28_1","unstructured":"I Herman. 2013. Eleven sparql 1.1 specifications are w3c recommendation. w3. org."},{"key":"e_1_2_2_29_1","unstructured":"Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings convolutional neural networks and incremental parsing. To appear"},{"key":"e_1_2_2_30_1","volume-title":"Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.","author":"Ilyas Andrew","year":"2022","unstructured":"Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. 2022. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622."},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Jitesh Jain Jiachen Li MangTik Chiu Ali Hassani Nikita Orlov and Humphrey Shi. 2022. OneFormer: One Transformer to Rule Universal Image Segmentation. arXiv.","DOI":"10.1109\/CVPR52729.2023.00292"},{"key":"e_1_2_2_32_1","first-page":"27","article-title":"Optimizing DNN computation with relaxed graph substitutions","volume":"1","author":"Jia Zhihao","year":"2019","unstructured":"Zhihao Jia, James Thomas, Todd Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2019. Optimizing DNN computation with relaxed graph substitutions. Proceedings of Machine Learning and Systems, 1 (2019), 27\u201339.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_2_33_1","volume-title":"Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, and Lucile Saulnier.","author":"Jiang Albert Q","year":"2023","unstructured":"Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, and Lucile Saulnier. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035933"},{"key":"e_1_2_2_35_1","volume-title":"NeurIPS MLSys Workshop. 3, 10","author":"Kang Daniel","year":"2018","unstructured":"Daniel Kang, Deepti Raghavan, Peter Bailis, and Matei Zaharia. 2018. Model assertions for debugging machine learning. In NeurIPS MLSys Workshop. 3, 10."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.1140"},{"key":"e_1_2_2_37_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11128\u201311137","author":"Kim Edward","year":"2020","unstructured":"Edward Kim, Divya Gopinath, Corina Pasareanu, and Sanjit A Seshia. 2020. A programmatic and semantic approach to explaining and debugging neural network based object detectors. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 11128\u201311137."},{"key":"e_1_2_2_38_1","volume-title":"International conference on machine learning. 1885\u20131894","author":"Koh Pang Wei","year":"2017","unstructured":"Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning. 1885\u20131894."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-3823"},{"key":"e_1_2_2_41_1","volume-title":"Meihui Zhang, Kaiping Zheng, Gang Chen, Beng Chin Ooi, and Wei Luen James Yip.","author":"Lee Chonho","year":"2017","unstructured":"Chonho Lee, Zhaojing Luo, Kee Yuan Ngiam, Meihui Zhang, Kaiping Zheng, Gang Chen, Beng Chin Ooi, and Wei Luen James Yip. 2017. Big healthcare data analytics: Challenges and applications. Handbook of large-scale distributed computing in smart healthcare, 11\u201341."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2009.08.006"},{"key":"e_1_2_2_43_1","first-page":"3475","article-title":"UNQOVERing Stereotyping Biases via Underspecified Questions","volume":"2020","author":"Li Tao","year":"2020","unstructured":"Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing Stereotyping Biases via Underspecified Questions. In Findings of the Association for Computational Linguistics: EMNLP 2020. 3475\u20133489.","journal-title":"Findings of the Association for Computational Linguistics: EMNLP"},{"key":"e_1_2_2_44_1","volume-title":"International Conference on Machine Learning. 6565\u20136576","author":"Liang Paul Pu","year":"2021","unstructured":"Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning. 6565\u20136576."},{"key":"e_1_2_2_45_1","volume-title":"Algorithms for Verifying Deep Neural Networks. Foundations and Trends\u00ae in Optimization, 4, 3-4","author":"Liu Changliu","year":"2021","unstructured":"Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J Kochenderfer. 2021. Algorithms for Verifying Deep Neural Networks. Foundations and Trends\u00ae in Optimization, 4, 3-4 (2021), 244\u2013404."},{"key":"e_1_2_2_46_1","doi-asserted-by":"crossref","unstructured":"Qing Lyu Shreya Havaldar Adam Stein Li Zhang Delip Rao Eric Wong Marianna Apidianaki and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379.","DOI":"10.18653\/v1\/2023.ijcnlp-main.20"},{"key":"e_1_2_2_47_1","volume-title":"Proceedings of the ACM on Programming Languages, 6, OOPSLA1","author":"Mariano Benjamin","year":"2022","unstructured":"Benjamin Mariano, Yanju Chen, Yu Feng, Greg Durrett, and I\u015fil Dillig. 2022. Automated transpilation of imperative to functional code using neural-guided program synthesis. Proceedings of the ACM on Programming Languages, 6, OOPSLA1 (2022), 1\u201327."},{"key":"e_1_2_2_48_1","volume-title":"International Conference on Machine Learning. 2430\u20132439","author":"Mirhoseini Azalia","year":"2017","unstructured":"Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. Device placement optimization with reinforcement learning. In International Conference on Machine Learning. 2430\u20132439."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","unstructured":"Aaditya Naik Adam Stein Yinjun Wu Mayur Naik and Eric Wong. 2024. TorchQL: A Programming Framework for Integrity Constraints in Machine Learning. https:\/\/doi.org\/10.5281\/zenodo.10723160 10.5281\/zenodo.10723160","DOI":"10.5281\/zenodo.10723160"},{"key":"e_1_2_2_50_1","unstructured":"Aaditya Naik Yinjun Wu Mayur Naik and Eric Wong. 2023. Do Machine Learning Models Learn Common Sense? arXiv preprint arXiv:2303.01433."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.154"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","unstructured":"The pandas development team. 2020. pandas-dev\/pandas: Pandas. https:\/\/doi.org\/10.5281\/zenodo.3509134 10.5281\/zenodo.3509134","DOI":"10.5281\/zenodo.3509134"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/130283.130294"},{"key":"e_1_2_2_54_1","volume-title":"Laurens Van Der Maaten, and Kilian Q Weinberger","author":"Pleiss Geoff","year":"2017","unstructured":"Geoff Pleiss, Danlu Chen, Gao Huang, Tongcheng Li, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Memory-efficient implementation of densenets. arXiv preprint arXiv:1707.06990."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41060-016-0027-9"},{"key":"e_1_2_2_57_1","unstructured":"Hadi Salman Saachi Jain Andrew Ilyas Logan Engstrom Eric Wong and Aleksander Madry. 2022. When does Bias Transfer in Transfer Learning? arXiv preprint arXiv:2207.02842."},{"key":"e_1_2_2_58_1","volume-title":"Proceedings of the 2019 International Conference on Management of Data. 1993\u20131996","author":"Schelter Sebastian","year":"2019","unstructured":"Sebastian Schelter, Felix Biessmann, Dustin Lange, Tammo Rukat, Phillipp Schmidt, Stephan Seufert, Pierre Brunelle, and Andrey Taptunov. 2019. Unit testing data with deequ. In Proceedings of the 2019 International Conference on Management of Data. 1993\u20131996."},{"key":"e_1_2_2_59_1","volume-title":"Predicting in-hospital mortality of icu patients: The physionet\/computing in cardiology challenge","author":"Silva Ikaro","year":"2012","unstructured":"Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. 2012. Predicting in-hospital mortality of icu patients: The physionet\/computing in cardiology challenge 2012. In 2012 Computing in Cardiology. 245\u2013248."},{"key":"e_1_2_2_60_1","unstructured":"Sahil Singla and Soheil Feizi. 2021. Salient ImageNet: How to discover spurious features in Deep Learning? arXiv preprint arXiv:2110.04301."},{"key":"e_1_2_2_61_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12853\u201312862","author":"Singla Sahil","year":"2021","unstructured":"Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. 2021. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12853\u201312862."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980983.2908102"},{"key":"e_1_2_2_63_1","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 9180\u20139211","author":"Smith Eric Michael","year":"2022","unstructured":"Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. \u201cI\u2019m sorry to hear that\u201d: Finding New Biases in Language Models with a Holistic Descriptor Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 9180\u20139211."},{"key":"e_1_2_2_64_1","volume-title":"Hashimoto","author":"Taori Rohan","year":"2023","unstructured":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https:\/\/github.com\/tatsu-lab\/stanford_alpaca"},{"key":"e_1_2_2_65_1","unstructured":"The NumPy Team. 2024. NumPy. https:\/\/numpy.org\/"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"volume-title":"Exploratory data analysis. 2","author":"Tukey John W","key":"e_1_2_2_67_1","unstructured":"John W Tukey. 1977. Exploratory data analysis. 2, Reading, MA."},{"key":"e_1_2_2_68_1","unstructured":"Daisuke Wakabayashi. 2018. Self-driving uber car kills pedestrian in Arizona where Robots Roam. https:\/\/www.nytimes.com\/2018\/03\/19\/technology\/uber-driverless-fatality.html"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178487.3178491"},{"key":"e_1_2_2_70_1","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_71_1","unstructured":"Benjamin Wilson Judy Hoffman and Jamie Morgenstern. 2019. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097."},{"key":"e_1_2_2_72_1","volume-title":"International Conference on Machine Learning. 11205\u201311216","author":"Wong Eric","year":"2021","unstructured":"Eric Wong, Shibani Santurkar, and Aleksander Madry. 2021. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning. 11205\u201311216."},{"key":"e_1_2_2_73_1","volume-title":"ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations.","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations."},{"key":"e_1_2_2_74_1","doi-asserted-by":"crossref","first-page":"e0195360","DOI":"10.1371\/journal.pone.0195360","article-title":"Time series smoother for effect detection","volume":"13","author":"You Cheng","year":"2018","unstructured":"Cheng You, Dennis KJ Lin, and S Stanley Young. 2018. Time series smoother for effect detection. PloS one, 13, 4 (2018), e0195360.","journal-title":"PloS one"},{"key":"e_1_2_2_75_1","volume-title":"2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10)","author":"Zaharia Matei","year":"2010","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10)."},{"key":"e_1_2_2_76_1","volume-title":"Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15, 11","author":"Zech John R","year":"2018","unstructured":"John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. 2018. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15, 11 (2018), e1002683."},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1323"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649841","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649841","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:06Z","timestamp":1750287246000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649841"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":77,"journal-issue":{"issue":"OOPSLA1","published-print":{"date-parts":[[2024,4,29]]}},"alternative-id":["10.1145\/3649841"],"URL":"https:\/\/doi.org\/10.1145\/3649841","relation":{},"ISSN":["2475-1421"],"issn-type":[{"type":"electronic","value":"2475-1421"}],"subject":[],"published":{"date-parts":[[2024,4,29]]},"assertion":[{"value":"2024-04-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}