{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T16:26:53Z","timestamp":1761582413284,"version":"build-2065373602"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","license":[{"start":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T00:00:00Z","timestamp":1759968000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CCF-1762299, CCF-1918889, CNS-1908304, CCF-1901376, CNS-2120696, CCF- 2210831, and CCF-2319471, CCF-2422130, CCF-2403211"],"award-info":[{"award-number":["CCF-1762299, CCF-1918889, CNS-1908304, CCF-1901376, CNS-2120696, CCF- 2210831, and CCF-2319471, CCF-2422130, CCF-2403211"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR00112590133"],"award-info":[{"award-number":["HR00112590133"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2025,10,9]]},"abstract":"<jats:p>The goal of active learning for program synthesis is to synthesize the desired program by asking targeted questions that minimize user interaction. While prior work has explored active learning in the purely symbolic setting, such techniques are inadequate for the increasingly popular paradigm of neurosymbolic program synthesis, where the synthesized program incorporates neural components. When applied to the neurosymbolic setting, such techniques can -- and, in practice, do -- return an unintended program due to mispredictions of neural components. This paper proposes a new active learning technique that can handle the unique challenges posed by neural network mispredictions. Our approach is based upon a new evaluation strategy called constrained conformal evaluation (CCE), which accounts for neural mispredictions while taking into account user-provided feedback. Our proposed method iteratively makes CCE more precise until all remaining programs are guaranteed to be observationally equivalent. We have implemented this method in a tool called SmartLabel and experimentally evaluated it on three neurosymbolic domains. Our results demonstrate that SmartLabel identifies the ground truth program for 98% of the benchmarks, requiring under 5 rounds of user interaction on average. In contrast, prior techniques for active learning are only able to converge to the ground truth program for at most 65% of the benchmarks.<\/jats:p>","DOI":"10.1145\/3763102","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T08:49:50Z","timestamp":1759999790000},"page":"1455-1483","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Active Learning for Neurosymbolic Program Synthesis"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7688-6133","authenticated-orcid":false,"given":"Celeste","family":"Barnaby","sequence":"first","affiliation":[{"name":"University of Texas at Austin, Austin, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4680-5157","authenticated-orcid":false,"given":"Qiaochu","family":"Chen","sequence":"additional","affiliation":[{"name":"New York University, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6175-6919","authenticated-orcid":false,"given":"Ramya","family":"Ramalingam","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9990-7566","authenticated-orcid":false,"given":"Osbert","family":"Bastani","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8006-1230","authenticated-orcid":false,"given":"I\u015f\u0131l","family":"Dillig","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Anastasios Angelopoulos Stephen Bates Jitendra Malik and Michael I Jordan. 2020. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193."},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Anastasios N Angelopoulos Stephen Bates et al. 2023. Conformal prediction: A gentle introduction. Foundations and Trends\u00ae in Machine Learning 16 4 (2023) 494\u2013591.","DOI":"10.1561\/2200000101"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3426470"},{"key":"e_1_2_1_4_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732."},{"key":"e_1_2_1_5_1","unstructured":"Vineeth Balasubramanian Shen-Shyang Ho and Vladimir Vovk. 2014. Conformal prediction for reliable machine learning: theory adaptations and applications. Newnes."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Celeste Barnaby Qiaochu Chen Ramya Ramalingam Osbert Bastani and Isil Dillig. 2025. Active Learning for Neurosymbolic Program Synthesis. https:\/\/doi.org\/10.48550\/arXiv.2508.15750 10.48550\/arXiv.2508.15750","DOI":"10.48550\/arXiv.2508.15750"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","unstructured":"Celeste Barnaby Qiaochu Chen Ramya Ramalingam Osbert Bastani and Isil Dillig. 2025. Artifact for \"Active Learning for Neurosymbolic Program Synthesis\". https:\/\/doi.org\/10.5281\/zenodo.16915436 10.5281\/zenodo.16915436","DOI":"10.5281\/zenodo.16915436"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3591248"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Celeste Barnaby Qiaochu Chen Chenglong Wang and Isil Dillig. 2024. PhotoScout: Synthesis-Powered Multi-Modal Image Search. arXiv preprint arXiv:2401.10464.","DOI":"10.1145\/3613904.3642319"},{"key":"e_1_2_1_10_1","first-page":"29362","article-title":"Practical adversarial multivalid conformal prediction","volume":"35","author":"Bastani Osbert","year":"2022","unstructured":"Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. 2022. Practical adversarial multivalid conformal prediction. Advances in Neural Information Processing Systems, 35 (2022), 29362\u201329373.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_11_1","unstructured":"Samuele Bortolotti Emanuele Marconato Tommaso Carraro Paolo Morettin Emile van Krieken Antonio Vergari Stefano Teso and Andrea Passerini. 2024. A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts. arXiv preprint arXiv:2406.10368."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1006\/inco.1995.1164"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/180139.181067"},{"key":"e_1_2_1_14_1","volume-title":"Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.","author":"Chan Chi-Min","year":"2023","unstructured":"Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Swarat Chaudhuri Kevin Ellis Oleksandr Polozov Rishabh Singh Armando Solar-Lezama Yisong Yue et al. 2021. Neurosymbolic programming. Foundations and Trends\u00ae in Programming Languages 7 3 (2021) 158\u2013243.","DOI":"10.1561\/2500000049"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3622863"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454047"},{"key":"e_1_2_1_18_1","unstructured":"Zhoujun Cheng Tianbao Xie Peng Shi Chengzu Li Rahul Nadkarni Yushi Hu Caiming Xiong Dragomir Radev Mari Ostendorf Luke Zettlemoyer et al. 2022. Binding language models in symbolic languages. arXiv preprint arXiv:2210.02875."},{"key":"e_1_2_1_19_1","volume-title":"Exact and Robust Conformal Inference Methods for Predictive Machine Learning with Dependent Data. In Conference On Learning Theory. 732\u2013749","author":"Chernozhukov Victor","year":"2018","unstructured":"Victor Chernozhukov, Kaspar W\u00fcthrich, and Zhu Yinchu. 2018. Exact and Robust Conformal Inference Methods for Predictive Machine Learning with Dependent Data. In Conference On Learning Theory. 732\u2013749."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/512950.512973"},{"key":"e_1_2_1_21_1","volume-title":"Analysis of a greedy active learning strategy. Advances in neural information processing systems, 17","author":"Dasgupta Sanjoy","year":"2004","unstructured":"Sanjoy Dasgupta. 2004. Analysis of a greedy active learning strategy. Advances in neural information processing systems, 17 (2004)."},{"key":"e_1_2_1_22_1","volume-title":"International conference on machine learning. 990\u2013998","author":"Devlin Jacob","year":"2017","unstructured":"Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i\/o. In International conference on machine learning. 990\u2013998."},{"key":"e_1_2_1_23_1","volume-title":"Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31","author":"Ellis Kevin","year":"2018","unstructured":"Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31 (2018)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140587.3062351"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72016-2_9"},{"key":"e_1_2_1_26_1","volume-title":"FOREST: An Interactive Multi-tree Synthesizer for Regular Expressions. In Tools and Algorithms for the Construction and Analysis of Systems","author":"Ferreira Margarida","year":"2021","unstructured":"Margarida Ferreira, Miguel Terra-Neves, Miguel Ventura, In\u00eas Lynce, and Ruben Martins. 2021. FOREST: An Interactive Multi-tree Synthesizer for Regular Expressions. In Tools and Algorithms for the Construction and Analysis of Systems, Jan Friso Groote and Kim Guldstrand Larsen (Eds.). Springer International Publishing, Cham. 152\u2013169. isbn:978-3-030-72016-2"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-77935-5_14"},{"key":"e_1_2_1_28_1","volume-title":"International Conference on Machine Learning. 1213\u20131222","author":"Gaunt Alexander L","year":"2017","unstructured":"Alexander L Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. 2017. Differentiable programs with neural libraries. In International Conference on Machine Learning. 1213\u20131222."},{"key":"e_1_2_1_29_1","volume-title":"Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34","author":"Gibbs Isaac","year":"2021","unstructured":"Isaac Gibbs and Emmanuel Candes. 2021. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34 (2021)."},{"key":"e_1_2_1_30_1","unstructured":"Isaac Gibbs and Emmanuel Cand\u00e8s. 2021. Conformal inference for online prediction with arbitrary distribution shifts. In NeurIPS."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the ACM on Programming Languages, 4, POPL","author":"Guo Zheng","year":"2019","unstructured":"Zheng Guo, Michael James, David Justo, Jiaxiao Zhou, Ziteng Wang, Ranjit Jhala, and Nadia Polikarpova. 2019. Program synthesis by type-guided abstraction refinement. Proceedings of the ACM on Programming Languages, 4, POPL (2019), 1\u201328."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01436"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3591285"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Shivam Handa and Martin Rinard. 2021. Program Synthesis Over Noisy Data with Guarantees. arXiv preprint arXiv:2103.05030.","DOI":"10.1145\/3368089.3409732"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409732"},{"key":"e_1_2_1_36_1","volume-title":"International Conference on Machine Learning. 4495\u20134506","author":"Huang Jiani","year":"2020","unstructured":"Jiani Huang, Calvin Smith, Osbert Bastani, Rishabh Singh, Aws Albarghouthi, and Mayur Naik. 2020. Generating programmatic referring expressions via program synthesis. In International Conference on Machine Learning. 4495\u20134506."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806833"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3586055"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3386025"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.747"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.325"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360602"},{"key":"e_1_2_1_43_1","volume-title":"International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJl07ySKvS","author":"Laich Larissa","year":"2020","unstructured":"Larissa Laich, Pavol Bielik, and Martin Vechev. 2020. Guiding Program Synthesis by Learning to Generate Examples. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJl07ySKvS"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_1_45_1","volume-title":"Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems, 31","author":"Manhaeve Robin","year":"2018","unstructured":"Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. 2018. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems, 31 (2018)."},{"key":"e_1_2_1_46_1","first-page":"72507","article-title":"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts","volume":"36","author":"Marconato Emanuele","year":"2023","unstructured":"Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. 2023. Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems, 36 (2023), 72507\u201372539.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3527315"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807459"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807459"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3632858"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428245"},{"key":"e_1_2_1_52_1","volume-title":"Dolphin: A programmable framework for scalable neurosymbolic learning. arXiv preprint arXiv:2410.03348.","author":"Naik Aaditya","year":"2024","unstructured":"Aaditya Naik, Jason Liu, Claire Wang, Amish Sethi, Saikat Dutta, Mayur Naik, and Eric Wong. 2024. Dolphin: A programmable framework for scalable neurosymbolic learning. arXiv preprint arXiv:2410.03348."},{"key":"e_1_2_1_53_1","volume-title":"NIPS workshop on deep learning and unsupervised feature learning.","author":"Netzer Yuval","year":"2011","unstructured":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning. 2011, 4."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2393347.2393363"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276520"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454063"},{"key":"e_1_2_1_57_1","unstructured":"Sangdon Park Edgar Dobriban Insup Lee and Osbert Bastani. 2021. PAC prediction sets under covariate shift. arXiv preprint arXiv:2106.09848."},{"key":"e_1_2_1_58_1","unstructured":"Ramya Ramalingam Sangdon Park and Osbert Bastani. 2024. Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction. arxiv:2405.15912."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2914770.2837671"},{"key":"e_1_2_1_60_1","volume-title":"A survey of deep active learning. ACM computing surveys (CSUR), 54, 9","author":"Ren Pengzhen","year":"2021","unstructured":"Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. A survey of deep active learning. ACM computing surveys (CSUR), 54, 9 (2021), 1\u201340."},{"key":"e_1_2_1_61_1","first-page":"6","article-title":"Less is more: Active learning with support vector machines","volume":"2","author":"Schohn Greg","year":"2000","unstructured":"Greg Schohn and David Cohn. 2000. Less is more: Active learning with support vector machines. In ICML. 2, 6.","journal-title":"ICML."},{"key":"e_1_2_1_62_1","unstructured":"Amazon Web Services. n.d.. Amazon Rekognition Documentation. https:\/\/docs.aws.amazon.com\/rekognition\/ Accessed: 2025-03-13"},{"key":"e_1_2_1_63_1","unstructured":"Burr Settles. 2009. Active learning literature survey."},{"key":"e_1_2_1_64_1","volume-title":"Journal of Machine Learning Research","author":"Shafer Glenn","year":"2008","unstructured":"Glenn Shafer and Vladimir Vovk. 2008. A tutorial on conformal prediction. Journal of Machine Learning Research, 9, Mar (2008), 371\u2013421."},{"key":"e_1_2_1_65_1","volume-title":"Learning differentiable programs with admissible neural heuristics. Advances in neural information processing systems, 33","author":"Shah Ameesh","year":"2020","unstructured":"Ameesh Shah, Eric Zhan, Jennifer Sun, Abhinav Verma, Yisong Yue, and Swarat Chaudhuri. 2020. Learning differentiable programs with admissible neural heuristics. Advances in neural information processing systems, 33 (2020), 4940\u20134952."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025153"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66706-5_18"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.298"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01092"},{"key":"e_1_2_1_70_1","volume-title":"Emmanuel Candes, and Aaditya Ramdas.","author":"Tibshirani Ryan J","year":"2019","unstructured":"Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. 2019. Conformal prediction under covariate shift. Advances in neural information processing systems, 32 (2019)."},{"key":"e_1_2_1_71_1","volume-title":"Houdini: Lifelong learning as program synthesis. Advances in neural information processing systems, 31","author":"Valkov Lazar","year":"2018","unstructured":"Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. 2018. Houdini: Lifelong learning as program synthesis. Advances in neural information processing systems, 31 (2018)."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/1706299.1706338"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485477"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485477"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3058738"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062365"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40231-1"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-96145-3_22"},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the ACM on Programming Languages, 2, POPL","author":"Wang Xinyu","year":"2017","unstructured":"Xinyu Wang, Isil Dillig, and Rishabh Singh. 2017. Program synthesis using abstraction refinement. Proceedings of the ACM on Programming Languages, 2, POPL (2017), 1\u201330."},{"key":"e_1_2_1_80_1","volume-title":"Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155.","author":"Wu Qingyun","year":"2023","unstructured":"Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155."},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"11569","author":"Xu Chen","year":"2021","unstructured":"Chen Xu and Yao Xie. 2021. Conformal prediction interval for dynamic time-series. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). PMLR, 11559\u201311569."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3591288"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3763102","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3763102","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:44:52Z","timestamp":1760031892000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3763102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,9]]},"references-count":82,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2025,10,9]]}},"alternative-id":["10.1145\/3763102"],"URL":"https:\/\/doi.org\/10.1145\/3763102","relation":{},"ISSN":["2475-1421"],"issn-type":[{"type":"electronic","value":"2475-1421"}],"subject":[],"published":{"date-parts":[[2025,10,9]]},"assertion":[{"value":"2025-03-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}