{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:14:55Z","timestamp":1772590495836,"version":"3.50.1"},"reference-count":177,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"name":"National Science Foundation","award":["IIS-1901386 and IIS-2402647"],"award-info":[{"award-number":["IIS-1901386 and IIS-2402647"]}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"crossref","award":["N00014-21-1-2707"],"award-info":[{"award-number":["N00014-21-1-2707"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100017005","name":"Allen Institute for Artificial Intelligence","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100017005","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>\n            LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Similarly, the more established techniques of crowdsourcing workflows decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space covers a designer\u2019s\n            <jats:italic>objectives<\/jats:italic>\n            and the\n            <jats:italic>tactics<\/jats:italic>\n            used to build workflows. We then surface\n            <jats:italic>strategies<\/jats:italic>\n            that mediate how workflows use tactics to achieve objectives. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify takeaways for effective chain design and raise implications for future research and development.\n          <\/jats:p>","DOI":"10.1145\/3716134","type":"journal-article","created":{"date-parts":[[2025,2,7]],"date-time":"2025-02-07T16:11:19Z","timestamp":1738944679000},"page":"1-57","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7290-068X","authenticated-orcid":false,"given":"Madeleine","family":"Grunde-McLaughlin","sequence":"first","affiliation":[{"name":"Paul G. Allen School of Computer Science &amp; Engineering, University of Washington, Seattle, Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3448-5961","authenticated-orcid":false,"given":"Michelle S.","family":"Lam","sequence":"additional","affiliation":[{"name":"Computer Science, Stanford University, Stanford, California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8784-2531","authenticated-orcid":false,"given":"Ranjay","family":"Krishna","sequence":"additional","affiliation":[{"name":"Paul G. Allen School of Computer Science &amp; Engineering, University of Washington, Seattle, Washington, USA and PRIOR, Allen Institute for Artificial Intelligence, Seattle, Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3255-0109","authenticated-orcid":false,"given":"Daniel S.","family":"Weld","sequence":"additional","affiliation":[{"name":"Semantic Scholar, Allen Institute for Artificial Intelligence, Seattle, Washington, USA and Paul G. Allen School of Computer Science &amp; Engineering, University of Washington, Seattle, Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6175-1655","authenticated-orcid":false,"given":"Jeffrey","family":"Heer","sequence":"additional","affiliation":[{"name":"Paul G. Allen School of Computer Science &amp; Engineering, University of Washington, Seattle, Washington, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,6,14]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"LangChain. 2024. Build a retrieval augmented generation (RAG) app. Retrieved from https:\/\/python.langchain.com\/v0.2\/docs\/tutorials\/rag\/"},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"1941","DOI":"10.1145\/3563657.3596001","volume-title":"Proceedings of the ACM Designing Interactive Systems Conference (DIS \u201923)","author":"Li-Yuan Chiou","year":"2023","unstructured":"Chiou Li-Yuan, Hung Peng-Kai, Liang Rung-Huei, and Wang Chun-Teng. 2023. Designing with AI: An exploration of co-ideation with image generators. In Proceedings of the ACM Designing Interactive Systems Conference (DIS \u201923). ACM, New York, NY, USA, 1941\u20131954."},{"key":"e_1_3_3_4_2","first-page":"2","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"3","author":"Agapie Elena","year":"2015","unstructured":"Elena Agapie, Jaime Teevan, and Andr\u00e9s Monroy-Hern\u00e1ndez. 2015. Crowdsourcing in the field: A case study using local crowds for event reporting. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3, 2\u201311."},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/2047196.2047203","volume-title":"Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology","author":"Ahmad Salman","year":"2011","unstructured":"Salman Ahmad, Alexis Battle, Zahan Malkani, and Sepander Kamvar. 2011. The Jabberwocky programming environment for structured social computing. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 53\u201364."},{"key":"e_1_3_3_6_2","volume-title":"A Pattern Language: Towns, Buildings, Construction","author":"Alexander Christopher","year":"1977","unstructured":"Christopher Alexander. 1977. A Pattern Language: Towns, Buildings, Construction. Oxford University Press."},{"key":"e_1_3_3_7_2","unstructured":"Garrett Allen Gaole He and Ujwal Gadiraju. 2023. Power-up! What Can Generative Models Do for Human Computation Workflows?"},{"key":"e_1_3_3_8_2","first-page":"13","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"8","author":"Alshaibani Abdullah","year":"2020","unstructured":"Abdullah Alshaibani, Sylvia Carrell, Li-Hsin Tseng, Jungmin Shin, and Alexander Quinn. 2020. Privacy-preserving face redaction using crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8, 13\u201322."},{"key":"e_1_3_3_9_2","first-page":"27","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"9","author":"Alshaibani Abdullah","year":"2021","unstructured":"Abdullah Alshaibani and Alexander J. Quinn. 2021. Pterodactyl: Two-step redaction of images for robust face deidentification. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 9, 27\u201334."},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1145\/2145204.2145382","volume-title":"Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work","author":"Ambati Vamshi","year":"2012","unstructured":"Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. 2012. Collaborative workflow for crowdsourcing translation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 1191\u20131194."},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1145\/2556288.2557158","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Andr\u00e9 Paul","year":"2014","unstructured":"Paul Andr\u00e9, Robert E. Kraut, and Aniket Kittur. 2014. Effects of simultaneous and sequential work structures on distributed collaborative interdependent tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 139\u2013148."},{"key":"e_1_3_3_12_2","first-page":"9","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"1","author":"Andr\u00e9 Paul","year":"2013","unstructured":"Paul Andr\u00e9, Haoqi Zhang, Juho Kim, Lydia Chilton, Steven Dow, and Robert Miller. 2013. Community clustering: Leveraging an academic crowd to form coherent conference sessions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 1, 9\u201316."},{"key":"e_1_3_3_13_2","first-page":"39","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Andreas Jacob","year":"2016","unstructured":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 39\u201348."},{"key":"e_1_3_3_14_2","first-page":"612","volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing","author":"Anya Obinna","year":"2015","unstructured":"Obinna Anya. 2015. Bridge the gap! What can work design in crowdwork learn from work design theories?. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, 612\u2013627."},{"key":"e_1_3_3_15_2","first-page":"1","volume-title":"Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology","author":"Arawjo Ian","year":"2023","unstructured":"Ian Arawjo, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: An open-source visual programming environment for prompt engineering. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 1\u20133."},{"key":"e_1_3_3_16_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Arora Simran","year":"2023","unstructured":"Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask me anything: A simple strategy for prompting language models. In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=bhUPJnS2g0X"},{"key":"e_1_3_3_17_2","first-page":"18","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922)","volume":"81","author":"Bae S. Sandra","year":"2022","unstructured":"S. Sandra Bae, Clement Zheng, Mary Etta West, Ellen Yi-Luen Do, Samuel Huron, and Danielle Albers Szafir. 2022. Making data tangible: A cross-disciplinary design space for data physicalization. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM, New York, NY, USA, Article 81, 18 pages. DOI: 10.1145\/3491102.3501939"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1145\/3442188.3445922","volume-title":"Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency","author":"Bender Emily M.","year":"2021","unstructured":"Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610\u2013623."},{"key":"e_1_3_3_19_2","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1145\/2047196.2047201","volume-title":"Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology","author":"Bernstein Michael S.","year":"2011","unstructured":"Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 33\u201342."},{"key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1145\/1866029.1866078","volume-title":"Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology","author":"Bernstein Michael S.","year":"2010","unstructured":"Michael S. Bernstein, Greg Little, Robert C. Miller, Bj\u00f6rn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A word processor with a crowd inside. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, 313\u2013322."},{"key":"e_1_3_3_21_2","first-page":"1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","author":"Bragg Danielle","year":"2021","unstructured":"Danielle Bragg, Naomi Caselli, John W. Gallagher, Miriam Goldberg, Courtney J. Oka, and William Thies. 2021. ASL sea battle: Gamifying sign language data collection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1\u201313."},{"key":"e_1_3_3_22_2","volume-title":"Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology","author":"Bragg Jonathan","year":"2018","unstructured":"Jonathan Bragg, Mausam, and Daniel S. Weld. 2018. Sprout: Crowd-powered task design for crowdsourcing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Retrieved from https:\/\/api.semanticscholar.org\/corpusid:51948377"},{"key":"e_1_3_3_23_2","first-page":"25","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"1","author":"Bragg Jonathan","year":"2013","unstructured":"Jonathan Bragg and Daniel Weld. 2013. Crowdsourcing multi-label classification for taxonomy creation. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 1, 25\u201333."},{"issue":"2","key":"e_1_3_3_24_2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77\u2013101.","journal-title":"Qualitative Research in Psychology"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","unstructured":"Victor S. Bursztyn David Demeter Doug Downey and Larry Birnbaum. 2022. Learning to perform complex tasks through compositional fine-tuning of language models. arXiv:2210.12607. Retrieved from 10.48550\/arXiv.2210.12607","DOI":"10.48550\/arXiv.2210.12607"},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1145\/3126594.3126640","volume-title":"Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology","author":"Butler Crystal","year":"2017","unstructured":"Crystal Butler, Stephanie Michalowicz, Lakshmi Subramanian, and Winslow Burleson. 2017. More than a feeling: The MiFace framework for defining facial communication mappings. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 773\u2013786."},{"key":"e_1_3_3_27_2","volume-title":"Sketching User Experiences: Getting the Design Right and the Right Design","author":"Buxton Bill","year":"2010","unstructured":"Bill Buxton. 2010. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann."},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"2334","DOI":"10.1145\/3025453.3026044","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Chang Joseph Chee","year":"2017","unstructured":"Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2334\u20132346."},{"key":"e_1_3_3_29_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Chen Quanze","year":"2019","unstructured":"Quanze Chen, Jonathan Bragg, Lydia B. Chilton, and Dan S. Weld. 2019. Cicero: Multi-turn, contextual argumentation for accurate crowdsourcing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1\u201314."},{"key":"e_1_3_3_30_2","first-page":"600","volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing","author":"Cheng Justin","year":"2015","unstructured":"Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid crowd-machine learning classifiers. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, 600\u2013611."},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","first-page":"4061","DOI":"10.1145\/2702123.2702146","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","author":"Cheng Justin","year":"2015","unstructured":"Justin Cheng, Jaime Teevan, Shamsi T. Iqbal, and Michael S. Bernstein. 2015. Break it down: A comparison of macro-and microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 4061\u20134064."},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1145\/2556288.2557375","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Chilton Lydia B.","year":"2014","unstructured":"Lydia B. Chilton, Juho Kim, Paul Andr\u00e9, Felicia Cordeiro, James A. Landay, Daniel S. Weld, Steven P. Dow, Robert C. Miller, and Haoqi Zhang. 2014. Frenzy: Collaborative data organization for creating conference sessions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1255\u20131264."},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"1999","DOI":"10.1145\/2470654.2466265","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Chilton Lydia B.","year":"2013","unstructured":"Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1999\u20132008."},{"key":"e_1_3_3_34_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Chilton Lydia B.","year":"2019","unstructured":"Lydia B. Chilton, Savvas Petridis, and Maneesh Agrawala. 2019. VisiBlends: A flexible workflow for visual blends. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1\u201314."},{"key":"e_1_3_3_35_2","first-page":"41","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"2","author":"Christoforaki Maria","year":"2014","unstructured":"Maria Christoforaki and Panagiotis Ipeirotis. 2014. Step: A scalable testing and evaluation platform. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 2, 41\u201349."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. arXiv:2110.14168. Retrieved from 10.48550\/arXiv.2110.14168","DOI":"10.48550\/arXiv.2110.14168"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","unstructured":"Antonia Creswell and Murray Shanahan. 2022. Faithful reasoning using large language models. arXiv:2208.14271. Retrieved from 10.48550\/arXiv.2208.14271","DOI":"10.48550\/arXiv.2208.14271"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","unstructured":"Antonia Creswell Murray Shanahan and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv:2205.09712. Retrieved from 10.48550\/arXiv.2205.09712","DOI":"10.48550\/arXiv.2205.09712"},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.artint.2013.06.002","article-title":"POMDP-based control of workflows for crowdsourcing","volume":"202","author":"Dai Peng","year":"2013","unstructured":"Peng Dai, Christopher H. Lin, and Daniel S. Weld. 2013. POMDP-based control of workflows for crowdsourcing. Artificial Intelligence 202 (2013), 52\u201385.","journal-title":"Artificial Intelligence"},{"key":"e_1_3_3_40_2","first-page":"1168","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"24","author":"Dai Peng","year":"2010","unstructured":"Peng Dai and Daniel Weld. 2010. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 24, 1168\u20131174."},{"key":"e_1_3_3_41_2","first-page":"135","volume-title":"Proceedings of the 11th ACM International Conference on Web Search and Data Mining","author":"Difallah Djellel","year":"2018","unstructured":"Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical Turk workers. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, 135\u2013143."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/1879831.1879836"},{"key":"e_1_3_3_43_2","first-page":"32","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"4","author":"Drapeau Ryan","year":"2016","unstructured":"Ryan Drapeau, Lydia Chilton, Jonathan Bragg, and Daniel Weld. 2016. Microtalk: Using argumentation to improve crowdsourcing accuracy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 4, 32\u201341."},{"key":"e_1_3_3_44_2","volume-title":"ICML 2024 Conference","author":"Du Yilun","year":"2024","unstructured":"Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In ICML 2024 Conference. OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=zj7YuTE4t8"},{"key":"e_1_3_3_45_2","volume-title":"37th Conference on Neural Information Processing Systems (NeurIPS 2023)","author":"Dziri Nouha","year":"2023","unstructured":"Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, et al. 2023. Faith and fate: Limits of transformers on compositionality. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/file\/deb3c28192f979302c157cb653c15e90-Paper-Conference.pdf"},{"key":"e_1_3_3_46_2","doi-asserted-by":"crossref","unstructured":"Nouha Dziri Sivan Milton Mo Yu Osmar R. Zaiane and Siva Reddy. 2022. On the origin of hallucinations in conversational models: Is it the datasets or the models? arXiv:2204.07931. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2204.07931","DOI":"10.18653\/v1\/2022.naacl-main.387"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1111\/jerd.13046"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","unstructured":"Jacob Eisenstein Daniel Andor Bernd Bohnet Michael Collins and David Mimno. 2022. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from a pretrained language model. arXiv:2210.02498. Retrieved from 10.48550\/arXiv.2210.02498","DOI":"10.48550\/arXiv.2210.02498"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1101\/2023.04.25.23288588"},{"issue":"1","key":"e_1_3_3_50_2","first-page":"145","article-title":"Semantic scholar","volume":"106","author":"Fricke Suzanne","year":"2018","unstructured":"Suzanne Fricke. 2018. Semantic scholar. Journal of the Medical Library Association 106, 1 (2018), 145.","journal-title":"Journal of the Medical Library Association"},{"key":"e_1_3_3_51_2","volume-title":"Design Patterns Elements of Reusable Object-Oriented Software","author":"Gamma Erich","year":"1994","unstructured":"Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns Elements of Reusable Object-Oriented Software. Addison-Wesley."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","unstructured":"Zelalem Gero Chandan Singh Hao Cheng Tristan Naumann Michel Galley Jianfeng Gao and Hoifung Poon. 2023. Self-verification improves few-shot clinical information extraction. arXiv:2306.00024. Retrieved from 10.48550\/arXiv.2306.00024","DOI":"10.48550\/arXiv.2306.00024"},{"key":"e_1_3_3_53_2","first-page":"19","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201922)","volume":"115","author":"Gordon Mitchell L.","year":"2022","unstructured":"Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM, New York, NY, USA, Article 115, 19 pages. DOI: 10.1145\/3491102.3502004"},{"key":"e_1_3_3_54_2","first-page":"52","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"4","author":"Goto Shinsuke","year":"2016","unstructured":"Shinsuke Goto, Toru Ishida, and Donghui Lin. 2016. Understanding crowdsourcing workflow: Modeling and optimizing iterative and parallel processes. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 4. 52\u201358."},{"key":"e_1_3_3_55_2","first-page":"31","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"6","author":"Gouravajhala Sai","year":"2018","unstructured":"Sai Gouravajhala, Jinyeong Yim, Karthik Desingh, Yanda Huang, Odest Jenkins, and Walter Lasecki. 2018. Eureca: Enhanced understanding of real environments via crowd assistance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 6, 31\u201340."},{"key":"e_1_3_3_56_2","first-page":"79","volume-title":"AISec \u201923: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security","author":"Greshake Kai","year":"2023","unstructured":"Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you\u2019ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In AISec \u201923: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 79\u201390. DOI: 10.1145\/3605764.3623985"},{"key":"e_1_3_3_57_2","first-page":"651","volume-title":"Proceedings of the 29th Annual Symposium on User Interface Software and Technology","author":"Guo Anhong","year":"2016","unstructured":"Anhong Guo, Xiang\u2019Anthony\u2019 Chen, Haoran Qi, Samuel White, Suman Ghosh, Chieko Asakawa, and Jeffrey P. Bigham. 2016. Vizlens: A robust and interactive screen reader for interfaces in the real world. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 651\u2013664."},{"key":"e_1_3_3_58_2","first-page":"5826","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Guo Anhong","year":"2017","unstructured":"Anhong Guo, Jeeeun Kim, Xiang\u2019Anthony\u2019 Chen, Tom Yeh, Scott E. Hudson, Jennifer Mankoff, and Jeffrey P. Bigham. 2017. Facade: Auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 5826\u20135838."},{"key":"e_1_3_3_59_2","first-page":"371","volume-title":"Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology","author":"Guo Anhong","year":"2019","unstructured":"Anhong Guo, Junhan Kong, Michael Rivera, Frank F. Xu, and Jeffrey P. Bigham. 2019. Statelens: A reverse engineering solution for making existing dynamic touchscreens accessible. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, 371\u2013385."},{"key":"e_1_3_3_60_2","first-page":"14953","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Gupta Tanmay","year":"2023","unstructured":"Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 14953\u201314962."},{"key":"e_1_3_3_61_2","doi-asserted-by":"crossref","first-page":"2258","DOI":"10.1145\/2858036.2858364","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems","author":"Hahn Nathan","year":"2016","unstructured":"Nathan Hahn, Joseph Chang, Ji Eun Kim, and Aniket Kittur. 2016. The knowledge accelerator: Big picture thinking in small pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2258\u20132270."},{"key":"e_1_3_3_62_2","doi-asserted-by":"crossref","unstructured":"Ari Holtzman Peter West Vered Shwartz Yejin Choi and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn\u2019t always right. arXiv:2104.08315. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2104.08315","DOI":"10.18653\/v1\/2021.emnlp-main.564"},{"key":"e_1_3_3_63_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/302979.303030","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Horvitz Eric","year":"1999","unstructured":"Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 159\u2013166."},{"key":"e_1_3_3_64_2","unstructured":"Cheng-Yu Hsieh Si-An Chen Chun-Liang Li Yasuhisa Fujii Alexander Ratner Chen-Yu Lee Ranjay Krishna and Tomas Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. arXiv:2308.00675. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.00675"},{"key":"e_1_3_3_65_2","unstructured":"Qing Huang Yishun Wu Zhenchang Xing He Jiang Yu Cheng and Huan Jin. 2023. Adaptive intellect unleashed: The feasibility of knowledge transfer in large language models. arXiv:2308.04788. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.04788"},{"key":"e_1_3_3_66_2","first-page":"1","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)","author":"Huang Qing","year":"2023","unstructured":"Qing Huang, Jiahui Zhu, Zhilong Li, Zhenchang Xing, Changjing Wang, and Xiwei Xu. 2023. PCR-chain: Partial code reuse assisted by hierarchical chaining of prompts on frozen copilot. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 1\u20135."},{"key":"e_1_3_3_67_2","unstructured":"Qing Huang Zhou Zou Zhenchang Xing Zhenkang Zuo Xiwei Xu and Qinghua Lu. 2023. AI chain on large language model for unsupervised control flow graph generation for statically-typed partial code. arXiv:2306.00757. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2306.00757"},{"key":"e_1_3_3_68_2","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Huang Ting-Hao","year":"2018","unstructured":"Ting-Hao Huang, Joseph Chee Chang, and Jeffrey P. Bigham. 2018. Evorus: A crowd-powered conversational assistant built to automate itself over time. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1\u201313."},{"key":"e_1_3_3_69_2","first-page":"62","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"3","author":"Huang Ting-Hao","year":"2015","unstructured":"Ting-Hao Huang, Walter Lasecki, and Jeffrey Bigham. 2015. Guardian: A crowd-powered spoken dialog system for web Apis. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3, 62\u201371."},{"key":"e_1_3_3_70_2","first-page":"71","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"5","author":"Huang Yi-Ching","year":"2017","unstructured":"Yi-Ching Huang, Jiunn-Chia Huang, Hao-Chuan Wang, and Jane Hsu. 2017. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 5, 71\u201378."},{"key":"e_1_3_3_71_2","first-page":"1","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems","author":"Huffaker Jordan S.","year":"2020","unstructured":"Jordan S. Huffaker, Jonathan K. Kummerfeld, Walter S. Lasecki, and Mark S. Ackerman. 2020. Crowdsourced detection of emotionally manipulative language. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1\u201314."},{"issue":"4","key":"e_1_3_3_72_2","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1207\/s15327051hci0104_2","article-title":"Direct manipulation interfaces","volume":"1","author":"Hutchins Edwin L.","year":"1985","unstructured":"Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. 1985. Direct manipulation interfaces. Human\u2013Computer Interaction 1, 4 (1985), 311\u2013338.","journal-title":"Human\u2013Computer Interaction"},{"key":"e_1_3_3_73_2","first-page":"8","volume-title":"Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA \u201922)","volume":"35","author":"Jiang Ellen","year":"2022","unstructured":"Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J. Cai. 2022. PromptMaker: Prompt-based prototyping with large language models. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA \u201922). ACM, New York, NY, USA, Article 35, 8 pages. DOI: 10.1145\/3491101.3503564"},{"key":"e_1_3_3_74_2","unstructured":"Martin Josifoski Lars Klein Maxime Peyrard Yifei Li Saibo Geng Julian Paul Schnitzler Yuxing Yao Jiheng Wei Debjit Paul and Robert West. 2023. Flows: Building blocks of reasoning and collaborating AI. arXiv:2308.01285. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.01285"},{"key":"e_1_3_3_75_2","first-page":"89","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"5","author":"Kaur Harmanpreet","year":"2017","unstructured":"Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey Bigham, Jaime Teevan, Ece Kamar, and Walter Lasecki. 2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 5, 89\u201398."},{"key":"e_1_3_3_76_2","first-page":"1018","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing","author":"Kim Joy","year":"2016","unstructured":"Joy Kim and Andres Monroy-Hernandez. 2016. Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 1018\u20131027."},{"key":"e_1_3_3_77_2","first-page":"4017","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Kim Juho","year":"2014","unstructured":"Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 4017\u20134026."},{"key":"e_1_3_3_78_2","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1145\/2998181.2998196","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing","author":"Kim Joy","year":"2017","unstructured":"Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, and Michael S Bernstein. 2017. Mechanical novel: Crowdsourcing complex work through reflection and revision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 233\u2013245."},{"key":"e_1_3_3_79_2","first-page":"115","volume-title":"Proceedings of the 2023 ACM Designing Interactive Systems Conference","author":"Kim Jeongyeon","year":"2023","unstructured":"Jeongyeon Kim, Sangho Suh, Lydia B. Chilton, and Haijun Xia. 2023. Metaphorian: Leveraging large language models to support extended metaphor creation for science writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference, 115\u2013135."},{"key":"e_1_3_3_80_2","first-page":"1","volume-title":"Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology","author":"Soo Kim Tae","year":"2023","unstructured":"Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. 2023. Cells, generators, and lenses: Design framework for object-oriented interaction with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 1\u201318."},{"key":"e_1_3_3_81_2","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1145\/2145204.2145357","volume-title":"Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work","author":"Kittur Aniket","year":"2012","unstructured":"Aniket Kittur, Susheel Khamkar, Paul Andr\u00e9, and Robert Kraut. 2012. CrowdWeaver: Visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 1033\u20131036."},{"key":"e_1_3_3_82_2","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1145\/2441776.2441923","volume-title":"Proceedings of the 2013 Conference on Computer Supported Cooperative Work","author":"Kittur Aniket","year":"2013","unstructured":"Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, 1301\u20131318."},{"key":"e_1_3_3_83_2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/2047196.2047202","volume-title":"Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology","author":"Kittur Aniket","year":"2011","unstructured":"Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 43\u201352."},{"key":"e_1_3_3_84_2","first-page":"79","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"6","author":"Kobayashi Masaki","year":"2018","unstructured":"Masaki Kobayashi, Hiromi Morita, Masaki Matsubara, Nobuyuki Shimizu, and Atsuyuki Morishima. 2018. An empirical study on short-and long-term effects of self-correction in crowdsourced microtasks. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 6, 79\u201387."},{"key":"e_1_3_3_85_2","unstructured":"Ranjay Krishna. 2019. EasyTurk: A wrapper for custom AMT tasks. Retrieved from https:\/\/github.com\/ranjaykrishna\/easyturk"},{"key":"e_1_3_3_86_2","unstructured":"Alex Krizhevsky et al. 2009. Learning multiple layers of features from tiny images."},{"key":"e_1_3_3_87_2","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1145\/2145204.2145354","volume-title":"Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work","author":"Kulkarni Anand","year":"2012","unstructured":"Anand Kulkarni, Matthew Can, and Bj\u00f6rn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 1003\u20131012."},{"key":"e_1_3_3_88_2","first-page":"112","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"2","author":"Kulkarni Anand","year":"2014","unstructured":"Anand Kulkarni, Prayag Narula, David Rolnitzky, and Nathan Kontny. 2014. Wish: Amplifying creative ability with expert crowds. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 2, 112\u2013120."},{"key":"e_1_3_3_89_2","first-page":"1369","volume-title":"Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency","author":"Lai Vivian","year":"2023","unstructured":"Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023. Towards a science of human-AI decision making: An overview of design space in empirical human-subject studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1369\u20131385."},{"key":"e_1_3_3_90_2","first-page":"24","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI \u201923)","volume":"741","author":"Lam Michelle S.","year":"2023","unstructured":"Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang, James A. Landay, and Michael S. Bernstein. 2023. Model sketching: Centering concepts in early-stage machine learning model design. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI \u201923). ACM, New York, NY, USA, Article 741, 24 pages. DOI: 10.1145\/3544548.3581290"},{"key":"e_1_3_3_91_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Lam Michelle S.","year":"2024","unstructured":"Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept induction: Analyzing unstructured text with high-level concepts using LLooM. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1\u201328."},{"key":"e_1_3_3_92_2","unstructured":"LangChain. 2022. LangChain. Retrieved from https:\/\/www.langchain.com\/"},{"key":"e_1_3_3_93_2","doi-asserted-by":"crossref","first-page":"1925","DOI":"10.1145\/2702123.2702565","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","author":"Lasecki Walter S.","year":"2015","unstructured":"Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced user interfaces that come to life as you sketch them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 1925\u20131934."},{"key":"e_1_3_3_94_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1145\/2501988.2502057","volume-title":"Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology","author":"Lasecki Walter S.","year":"2013","unstructured":"Walter S. Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F. Allen, and Jeffrey P. Bigham. 2013. Chorus: A crowd-powered conversational assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, 151\u2013162."},{"key":"e_1_3_3_95_2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/2642918.2647349","volume-title":"Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology","author":"LaToza Thomas D.","year":"2014","unstructured":"Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and Andr\u00e9 Van Der Hoek. 2014. Microtask programming: Building software with a crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 43\u201354."},{"key":"e_1_3_3_96_2","first-page":"817","volume-title":"Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology","author":"Lee Sang Won","year":"2017","unstructured":"Sang Won Lee, Yujin Zhang, Isabelle Wong, Yiwei Yang, Stephanie D. O\u2019Keefe, and Walter S. Lasecki. 2017. Sketchexpress: Remixing animations for more effective crowd-powered prototyping of interactive interfaces. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 817\u2013828."},{"key":"e_1_3_3_97_2","unstructured":"Yoav Levine Itay Dalmedigos Ori Ram Yoel Zeldes Daniel Jannai Dor Muhlgay Yoni Osin Opher Lieber Barak Lenz Shai Shalev-Shwartz et al. 2022. Standing on the shoulders of giant frozen language models. arXiv:2204.10019. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2204.10019"},{"key":"e_1_3_3_98_2","unstructured":"Cheng Li Mingyang Zhang Qiaozhu Mei Yaqing Wang Spurthi Amba Hombaiah Yi Liang and Michael Bendersky. 2023. Teach LLMs to personalize\u2013An approach inspired by writing education. arXiv:2308.07968. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.07968"},{"key":"e_1_3_3_99_2","first-page":"18","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923)","volume":"2264","author":"Li Guohao","year":"2024","unstructured":"Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2024. CAMEL: Communicative agents for \u201cmind\u201d exploration of large language model society. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923). Curran Associates Inc., Red Hook, NY, USA, Article 2264, 18 pages."},{"key":"e_1_3_3_100_2","volume-title":"ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Li Jiyi","year":"2024","unstructured":"Jiyi Li. 2024. A comparative study on annotation quality of crowdsourcing and LLM via label aggregation. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. DOI: 10.1109\/ICASSP48485.2024.10447803"},{"key":"e_1_3_3_101_2","doi-asserted-by":"crossref","unstructured":"Tian Liang Zhiwei He Wenxiang Jiao Xing Wang Yan Wang Rui Wang Yujiu Yang Zhaopeng Tu and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv:2305.19118. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2305.19118","DOI":"10.18653\/v1\/2024.emnlp-main.992"},{"key":"e_1_3_3_102_2","first-page":"37","volume-title":"Proceedings of the 11th AAAI Conference on Human Computation","author":"Liem Beatrice","year":"2011","unstructured":"Beatrice Liem, Haoqi Zhang, and Yiling Chen. 2011. An iterative dual pathway structure for speech-to-text transcription. In Proceedings of the 11th AAAI Conference on Human Computation. AAAI Press, 37\u201342."},{"key":"e_1_3_3_103_2","first-page":"143","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"2","author":"Lin Chi-Chin","year":"2014","unstructured":"Chi-Chin Lin, Yi-Ching Huang, and Jane Yung-jen Hsu. 2014. Crowdsourced explanations for humorous internet memes based on linguistic theories. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 2, 143\u2013150."},{"key":"e_1_3_3_104_2","first-page":"740","volume-title":"Proceedings of the 13th European Conference on Computer Vision (ECCV \u201924)","volume":"13","author":"Lin Tsung-Yi","year":"2014","unstructured":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proceedings of the 13th European Conference on Computer Vision (ECCV \u201924), Part V 13. Springer, 740\u2013755."},{"key":"e_1_3_3_105_2","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1145\/1837885.1837907","volume-title":"Proceedings of the ACM SIGKDD Workshop on Human Computation","author":"Little Greg","year":"2010","unstructured":"Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 68\u201376."},{"key":"e_1_3_3_106_2","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1145\/1866029.1866040","volume-title":"Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology","author":"Little Greg","year":"2010","unstructured":"Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. Turkit: Human computation algorithms on mechanical Turk. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, 57\u201366."},{"key":"e_1_3_3_107_2","doi-asserted-by":"crossref","unstructured":"Angli Liu Stephen Soderland Jonathan Bragg C. H. Lin Xiao Ling and Daniel S. Weld. 2016. Effective crowd annotation for relation extraction. In North American Chapter of the Association for Computational Linguistics. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:10705630","DOI":"10.18653\/v1\/N16-1104"},{"key":"e_1_3_3_108_2","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Liu Ching","year":"2018","unstructured":"Ching Liu, Juho Kim, and Hao-Chuan Wang. 2018. ConceptScape: Collaborative concept mapping for video learning. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1\u201312."},{"key":"e_1_3_3_109_2","unstructured":"Ryan Liu Howard Yen Raja Marjieh Thomas L. Griffiths and Ranjay Krishna. 2023. Improving interpersonal communication by simulating audiences with language models. arXiv:2311.00687. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2311.00687"},{"key":"e_1_3_3_110_2","first-page":"1","volume-title":"Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology","author":"Liu Vivian","year":"2022","unstructured":"Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 1\u201317."},{"key":"e_1_3_3_111_2","unstructured":"Tao Long Dorothy Zhang Grace Li Batool Taraif Samia Menon Kynnedy Simone Smith Sitong Wang Katy Ilonka Gero and Lydia B Chilton. 2023. Tweetorial hooks: Generative AI tools to motivate science on social media. arXiv:2305.12265. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2305.12265"},{"key":"e_1_3_3_112_2","first-page":"110","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"3","author":"Luther Kurt","year":"2015","unstructured":"Kurt Luther, Nathan Hahn, Steven Dow, and Aniket Kittur. 2015. Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 3, 110\u2013119."},{"key":"e_1_3_3_113_2","unstructured":"Stephen MacNeil Andrew Tran Joanne Kim Ziheng Huang Seth Bernstein and Dan Mogil. 2023. Prompt middleware: Mapping prompts for large language models to UI affordances. arXiv:2307.01142. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2307.01142"},{"key":"e_1_3_3_114_2","first-page":"131000","volume-title":"Advances in Neural Information Processing Systems","author":"Aggarwal Pranjal","year":"2024","unstructured":"Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui and Mausam. 2024. AutoMix: Automatically Mixing Language Models. In Advances in Neural Information Processing Systems. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.). Curran Associates, Inc, 131000\u2013131034. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2024\/file\/ecda225cb187b40ea8edc1f46b03ffda-Paper-Conference.pdf"},{"key":"e_1_3_3_115_2","first-page":"46534","article-title":"Self-refine: Iterative refinement with self-feedback","volume":"36","author":"Madaan Aman","year":"2024","unstructured":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, Vol. 36, 46534\u201346594.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_116_2","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Mahyar Narges","year":"2018","unstructured":"Narges Mahyar, Michael R. James, Michelle M. Ng, Reginald A. Wu, and Steven P. Dow. 2018. CommunityCrit: Inviting the public to improve and evaluate urban design ideas through micro-activities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1\u201314."},{"key":"e_1_3_3_117_2","unstructured":"Jiayuan Mao Chuang Gan Pushmeet Kohli Joshua B. Tenenbaum and Jiajun Wu. 2019. The neuro-symbolic concept learner: Interpreting scenes words and sentences from natural supervision. arXiv:1904.12584. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1904.12584"},{"key":"e_1_3_3_118_2","volume-title":"Proceedings of the 38th International Conference on Very Large Data Bases","author":"Marcus Adam","year":"2011","unstructured":"Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. 2011. Human-powered sorts and joins. In Proceedings of the 38th International Conference on Very Large Data Bases."},{"key":"e_1_3_3_119_2","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1145\/1989323.1989486","volume-title":"Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data","author":"Marcus Adam","year":"2011","unstructured":"Adam Marcus, Eugene Wu, David R. Karger, Samuel Madden, and Robert C. Miller. 2011. Demonstration of Qurk: A query processor for humanoperators. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, 1315\u20131318."},{"key":"e_1_3_3_120_2","doi-asserted-by":"crossref","unstructured":"Joshua Maynez Shashi Narayan Bernd Bohnet and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv:2005.00661.","DOI":"10.18653\/v1\/2020.acl-main.173"},{"key":"e_1_3_3_121_2","first-page":"139","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"4","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? collecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 4, 139\u2013148. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2005.00661"},{"key":"e_1_3_3_122_2","first-page":"1","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","author":"Mirowski Piotr","year":"2023","unstructured":"Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1\u201334."},{"key":"e_1_3_3_123_2","first-page":"86","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"7","author":"Mohanty Vikram","year":"2019","unstructured":"Vikram Mohanty, Kareem Abdol-Hamid, Courtney Ebersohl, and Kurt Luther. 2019. Second opinion: Supporting last-mile person identification with crowdsourcing and face recognition. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7, 86\u201396."},{"key":"e_1_3_3_124_2","doi-asserted-by":"crossref","unstructured":"Varun Nair Elliot Schumacher and Anitha Kannan. 2023. Generating medically-accurate summaries of patient-provider dialogue: A multi-stage approach using large language models. arXiv:2305.05982. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2305.05982","DOI":"10.18653\/v1\/2023.clinicalnlp-1.26"},{"key":"e_1_3_3_125_2","doi-asserted-by":"crossref","unstructured":"Varun Nair Elliot Schumacher Geoffrey Tso and Anitha Kannan. 2023. DERA: Enhancing large language model completions with dialog-enabled resolving agents. arXiv:2303.17071. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2303.17071","DOI":"10.18653\/v1\/2024.clinicalnlp-1.12"},{"key":"e_1_3_3_126_2","doi-asserted-by":"crossref","unstructured":"Shashi Narayan Shay B. Cohen and Mirella Lapata. 2018. Don\u2019t give me the details just the summary! Topic-aware convolutional neural networks for extreme summarization. arXiv:1808.08745. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1808.08745","DOI":"10.18653\/v1\/D18-1206"},{"key":"e_1_3_3_127_2","doi-asserted-by":"crossref","first-page":"3834","DOI":"10.1145\/2858036.2858169","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems","author":"Nebeling Michael","year":"2016","unstructured":"Michael Nebeling, Alexandra To, Anhong Guo, Adrian A. de Freitas, Jaime Teevan, Steven P. Dow, and Jeffrey P. Bigham. 2016. WearWrite: Crowd-assisted writing from smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 3834\u20133846."},{"key":"e_1_3_3_128_2","unstructured":"Feng Nie Meixi Chen Zhirui Zhang and Xu Cheng. 2022. Improving few-shot performance of language models via nearest neighbor calibration. arXiv:2212.02216. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2212.02216"},{"key":"e_1_3_3_129_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_130_2","unstructured":"Aditya G. Parameswaran Shreya Shankar Parth Asawa Naman Jain and Yujie Wang. 2023. Revisiting prompt engineering via declarative crowdsourcing. arXiv:2308.03854. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.03854"},{"key":"e_1_3_3_131_2","first-page":"1","volume-title":"Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology","author":"Park Joon Sung","year":"2022","unstructured":"Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 1\u201318."},{"key":"e_1_3_3_132_2","first-page":"2642918","volume-title":"Proceedings of the UIST","volume":"10","author":"Pavel Amy","year":"2014","unstructured":"Amy Pavel, Colorado Reed, Bj\u00f6rn Hartmann, and Maneesh Agrawala. 2014. Video digests: A browsable, skimmable format for informational lecture videos. In Proceedings of the UIST, Vol. 10. Citeseer, 2642918\u20132647400."},{"key":"e_1_3_3_133_2","first-page":"1291","volume-title":"Proceedings of the 16th ACM International Conference on Web Search and Data Mining","author":"Peris Charith","year":"2023","unstructured":"Charith Peris, Christophe Dupuy, Jimit Majmudar, Rahil Parikh, Sami Smaili, Richard Zemel, and Rahul Gupta. 2023. Privacy in the time of language models. In Proceedings of the 16th ACM International Conference on Web Search and Data Mining, 1291\u20131292."},{"key":"e_1_3_3_134_2","first-page":"121","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"1","author":"Pietrowicz Mary","year":"2013","unstructured":"Mary Pietrowicz, Danish Chopra, Amin Sadeghi, Puneet Chandra, Brian Bailey, and Karrie Karahalios. 2013. CrowdBand: An Automated Crowdsourcing Sound Composition System. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 1, 121\u2013129."},{"key":"e_1_3_3_135_2","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1109\/CVPR.2009.5206537","volume-title":"Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Quattoni Ariadna","year":"2009","unstructured":"Ariadna Quattoni and Antonio Torralba. 2009. Recognizing indoor scenes. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 413\u2013420."},{"key":"e_1_3_3_136_2","unstructured":"Justin Reppert Ben Rachbach Charlie George Luke Stebbing Jungwon Byun Maggie Appleton and Andreas Stuhlm\u00fcller. 2023. Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes. arXiv:2301.01751. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2301.01751"},{"key":"e_1_3_3_137_2","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1145\/2642918.2647409","volume-title":"Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology","author":"Retelny Daniela","year":"2014","unstructured":"Daniela Retelny, S\u00e9bastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 75\u201385."},{"key":"e_1_3_3_138_2","doi-asserted-by":"crossref","first-page":"1890","DOI":"10.1145\/2998181.2998332","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing","author":"Salehi Niloufar","year":"2017","unstructured":"Niloufar Salehi, Jaime Teevan, Shamsi Iqbal, and Ece Kamar. 2017. Communicating context to the crowd for complex writing tasks. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 1890\u20131901."},{"key":"e_1_3_3_139_2","first-page":"147","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"5","author":"Salisbury Elliot","year":"2017","unstructured":"Elliot Salisbury, Ece Kamar, and Meredith Morris. 2017. Toward scalable social alt text: Conversational crowdsourcing as a tool for refining vision-to-language technology for the blind. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 5, 147\u2013156."},{"key":"e_1_3_3_140_2","unstructured":"Timo Schick Jane Dwivedi-Yu Zhengbao Jiang Fabio Petroni Patrick Lewis Gautier Izacard Qingfei You Christoforos Nalmpantis Edouard Grave and Sebastian Riedel. 2022. Peer: A collaborative language model. arXiv:2208.11663. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2208.11663"},{"key":"e_1_3_3_141_2","unstructured":"Shreya Shankar Aditya G. Parameswaran and Eugene Wu. 2024. DocETL: Agentic query rewriting and evaluation for complex document processing. arXiv:2410.12189. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2410.12189"},{"key":"e_1_3_3_142_2","volume-title":"38th Conference on Neural Information Processing Systems (NeurIPS 2024)","author":"Shao Rulin","year":"2024","unstructured":"Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. 2024. Scaling retrieval-based language models with a trillion-token datastore. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2024\/file\/a5d8aba27dfef4e849e8cb03fb87a954-Paper-Conference.pdf"},{"key":"e_1_3_3_143_2","first-page":"1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","author":"Shi Yang","year":"2021","unstructured":"Yang Shi, Xingyu Lan, Jingwen Li, Zhaorui Li, and Nan Cao. 2021. Communicating with motion: A design space for animated visual narratives in data videos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1\u201313."},{"issue":"3","key":"e_1_3_3_144_2","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1080\/01449298208914450","article-title":"The future of interactive systems and the emergence of direct manipulation","volume":"1","author":"Shneiderman Ben","year":"1982","unstructured":"Ben Shneiderman. 1982. The future of interactive systems and the emergence of direct manipulation. Behaviour & Information Technology 1, 3 (1982), 237\u2013256.","journal-title":"Behaviour & Information Technology"},{"key":"e_1_3_3_145_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2024.2405782"},{"key":"e_1_3_3_146_2","first-page":"22323","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Stretcu Otilia","year":"2023","unstructured":"Otilia Stretcu, Edward Vendrow, Kenji Hata, Krishnamurthy Viswanathan, Vittorio Ferrari, Sasan Tavakkol, Wenlei Zhou, Aditya Avinash, Emming Luo, Neil Gordon Alldrin, et al. 2023. Agile modeling: From concept to classifier in minutes. In Proceedings of the IEEE\/CVF International Conference on Computer Vision, 22323\u201322334."},{"key":"e_1_3_3_147_2","doi-asserted-by":"crossref","unstructured":"D\u00eddac Sur\u00eds Sachit Menon and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. Retrieved from https:\/\/openaccess.thecvf.com\/content\/ICCV2023\/papers\/Suris_ViperGPT_Visual_Inference_via_Python_Execution_for_Reasoning_ICCV_2023_paper.pdf","DOI":"10.1109\/ICCV51070.2023.01092"},{"key":"e_1_3_3_148_2","doi-asserted-by":"crossref","first-page":"2657","DOI":"10.1145\/2858036.2858108","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems","author":"Teevan Jaime","year":"2016","unstructured":"Jaime Teevan, Shamsi T. Iqbal, and Curtis Von Veh. 2016. Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2657\u20132668."},{"key":"e_1_3_3_149_2","unstructured":"Christian Terwiesch and Karl Ulrich. 2023. M.B.A. Students vs. ChatGPT: Who comes up with more innovative ideas? The Wall Street Journal. Retrieved from https:\/\/www.wsj.com\/tech\/ai\/mba-students-vs-chatgpt-innovation-679edf3b"},{"key":"e_1_3_3_150_2","first-page":"17553","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Toubal Imad Eddine","year":"2024","unstructured":"Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin, Jan Dlabal, Wenlei Zhou, Enming Luo, Otilia Stretcu, Hao Xiong, Chun-Ta Lu, Howard Zhou, et al. 2024. Modeling collaborator: Enabling subjective vision classification with minimal human effort via LLM tool-use. In Proceedings of the IEEE\/CVF International Conference on Computer Vision, 17553\u201317563."},{"key":"e_1_3_3_151_2","doi-asserted-by":"crossref","first-page":"3523","DOI":"10.1145\/3025453.3025811","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Valentine Melissa A.","year":"2017","unstructured":"Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017. Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 3523\u20133537."},{"key":"e_1_3_3_152_2","first-page":"1","volume-title":"CHI \u201924: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems","author":"Wadinambiarachchi Samangi","year":"2024","unstructured":"Samangi Wadinambiarachchi, Ryan M. Kelly, Saumya Pareek, Qiushi Zhou, and Eduardo Velloso. 2024. The effects of generative AI on design fixation and divergent thinking. In CHI \u201924: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Article 380, Pages 1\u201318. DOI: 10.1145\/3613904.3642919"},{"key":"e_1_3_3_153_2","unstructured":"Angelina Wang Jamie Morgenstern and John P. Dickerson. 2024. Large language models cannot replace human participants because they cannot portray identity groups. arXiv:2402.01908. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2402.01908"},{"key":"e_1_3_3_154_2","doi-asserted-by":"crossref","unstructured":"Boshi Wang Xiang Deng and Huan Sun. 2022. Iteratively prompt pre-trained language models for chain of thought. arXiv:2203.08383. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2203.08383","DOI":"10.18653\/v1\/2022.emnlp-main.174"},{"key":"e_1_3_3_155_2","first-page":"1","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","author":"Wang Sitong","year":"2023","unstructured":"Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B. Chilton. 2023. PopBlends: Strategies for conceptual blending with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1\u201319."},{"key":"e_1_3_3_156_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_157_2","first-page":"405","volume-title":"Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work Social Computing","author":"Weir Sarah","year":"2015","unstructured":"Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing subgoal labels for how-to videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work Social Computing. ACM, New York, NY, 405\u2013416. DOI: 10.1145\/2675133.2675219"},{"key":"e_1_3_3_158_2","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1145\/2207676.2207709","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Willett Wesley","year":"2012","unstructured":"Wesley Willett, Jeffrey Heer, and Maneesh Agrawala. 2012. Strategies for crowdsourcing social data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 227\u2013236."},{"key":"e_1_3_3_159_2","unstructured":"Qingyun Wu Gagan Bansal Jieyu Zhang Yiran Wu Shaokun Zhang Erkang Zhu Beibin Li Li Jiang Xiaoyun Zhang and Chi Wang. 2023. AutoGen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv:2308.08155. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2308.08155"},{"key":"e_1_3_3_160_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts","author":"Wu Tongshuang","year":"2022","unstructured":"Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J. Cai. 2022. Prompt chainer: Chaining large language model prompts through visual programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts, 1\u201310."},{"key":"e_1_3_3_161_2","first-page":"1","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems","author":"Wu Tongshuang","year":"2022","unstructured":"Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 1\u201322."},{"key":"e_1_3_3_162_2","unstructured":"Tongshuang Wu Haiyi Zhu Maya Albayrak Alexis Axon Amanda Bertsch Wenxing Deng Ziqi Ding Bill Guo Sireesh Gururaja Tzu-Sheng Kuo et al. 2023. LLMs as workers in human-computational algorithms? Replicating crowdsourcing pipelines with LLMs. arXiv:2307.10168."},{"key":"e_1_3_3_163_2","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems","author":"Wu Tongshuang Sherry","year":"2021","unstructured":"Tongshuang Sherry Wu, Michael Terry, and Carrie J. Cai. 2021. AI Chains: Transparent and controllable human-AI interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. DOI: 10.1145\/3491102.3517582"},{"key":"e_1_3_3_164_2","volume-title":"Proceedings of the ICLR 2024 Workshop on Large Language Model (LLM) Agents","author":"Wu Yiran","year":"2024","unstructured":"Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2024. MathChat: Converse to tackle challenging math problems with LLM agents. In Proceedings of the ICLR 2024 Workshop on Large Language Model (LLM) Agents."},{"key":"e_1_3_3_165_2","volume-title":"37th Conference on Neural Information Processing Systems (NeurIPS 2023)","author":"Xie Yuxi","year":"2023","unstructured":"Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Qizhe Xie. 2023. Self-evaluation guided beam search for reasoning. In 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/file\/81fde95c4dc79188a69ce5b24d63010b-Paper-Conference.pdf"},{"key":"e_1_3_3_166_2","unstructured":"Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou and Xinyun Chen. 2023. Large language models as optimizers. arXiv:2309.03409. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2309.03409"},{"key":"e_1_3_3_167_2","doi-asserted-by":"crossref","unstructured":"Kevin Yang Nanyun Peng Yuandong Tian and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. arXiv:2210.06774. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2210.06774","DOI":"10.18653\/v1\/2022.emnlp-main.296"},{"key":"e_1_3_3_168_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201919)","author":"Yang Qian","year":"2019","unstructured":"Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T. Iqbal, and Jaime Teevan. 2019. Sketching NLP: A case study of exploring the right things to design with language intelligence. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201919). ACM, New York, NY, USA, 1\u201312. DOI: 10.1145\/3290605.3300415"},{"key":"e_1_3_3_169_2","first-page":"11809","article-title":"Tree of thoughts: Deliberate problem solving with large language models","volume":"36","author":"Yao Shunyu","year":"2024","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, Vol. 36, 11809\u201311822.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_170_2","first-page":"1220","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies","author":"Zaidan Omar","year":"2011","unstructured":"Omar Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, 1220\u20131229."},{"key":"e_1_3_3_171_2","first-page":"1","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","author":"Zamfirescu-Pereira J. D.","year":"2023","unstructured":"J. D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can\u2019t prompt: How non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1\u201321."},{"key":"e_1_3_3_172_2","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1145\/2207676.2207708","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Zhang Haoqi","year":"2012","unstructured":"Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 217\u2013226."},{"key":"e_1_3_3_173_2","unstructured":"Jieyu Zhang Ranjay Krishna Ahmed H. Awadallah and Chi Wang. 2023. EcoAssistant: Using LLM assistant more affordably and accurately. arXiv:2310.03046. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2310.03046"},{"key":"e_1_3_3_174_2","first-page":"1","volume-title":"UIST \u201923: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology","author":"Zhang Zheng","year":"2023","unstructured":"Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A human-AI argumentative writing assistant with visual programming and rapid draft prototyping. In UIST \u201923: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, Article 5, 1\u201330. DOI: 10.1145\/3586183.3606800"},{"key":"e_1_3_3_175_2","unstructured":"Zhuosheng Zhang Aston Zhang Mu Li and Alex Smola. 2022. Automatic chain of thought prompting in large language models. Retrieved from https:\/\/www.chatgpthero.io\/wp-content\/uploads\/2023\/12\/2210.03493.pdf"},{"key":"e_1_3_3_176_2","first-page":"599","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 3: System Demonstrations","author":"Zharikova Diliara","year":"2023","unstructured":"Diliara Zharikova, Daniel Kornev, Fedor Ignatov, Maxim Talimanchuk, Dmitry Evseev, Ksenya Petukhova, Veronika Smilga, Dmitry Karpov, Yana Shishkina, Dmitry Kosenko, et al. 2023. DeepPavlov dream: Platform for building generative AI assistants. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 3: System Demonstrations, 599\u2013607."},{"key":"e_1_3_3_177_2","unstructured":"Denny Zhou Nathanael Sch\u00e4rli Le Hou Jason Wei Nathan Scales Xuezhi Wang Dale Schuurmans Claire Cui Olivier Bousquet Quoc Le et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2205.10625"},{"key":"e_1_3_3_178_2","unstructured":"Yongchao Zhou Andrei Ioan Muresanu Ziwen Han Keiran Paster Silviu Pitis Harris Chan and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In ICLR. OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=92gvk82DE-"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3716134","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,14]],"date-time":"2025-06-14T16:17:12Z","timestamp":1749917832000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3716134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,14]]},"references-count":177,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3716134"],"URL":"https:\/\/doi.org\/10.1145\/3716134","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,14]]},"assertion":[{"value":"2024-05-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}