{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T07:57:36Z","timestamp":1776931056382,"version":"3.51.2"},"publisher-location":"New York, NY, USA","reference-count":85,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,3,23]]},"DOI":"10.1145\/3742413.3789105","type":"proceedings-article","created":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T11:32:24Z","timestamp":1772537544000},"page":"617-630","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-5472-282X","authenticated-orcid":false,"given":"Annalisa","family":"Szymanski","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5773-8239","authenticated-orcid":false,"given":"Oghenemaro","family":"Anuyah","sequence":"additional","affiliation":[{"name":"Microsoft Corporation, Redmond, Washington, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7902-7625","authenticated-orcid":false,"given":"Toby Jia-Jun","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2206-1720","authenticated-orcid":false,"given":"Ronald","family":"Metoyer","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,3,22]]},"reference":[{"key":"e_1_3_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Muhammad\u00a0Aurangzeb Ahmad Ilker Yaramis and Taposh\u00a0Dutta Roy. 2023. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2311.01463 (2023).","DOI":"10.20944\/preprints202310.1662.v1"},{"key":"e_1_3_3_1_3_2","unstructured":"Khairul Alam Kartik Mittal Banani Roy and Chanchal Roy. 2024. Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2411.10873 (2024)."},{"key":"e_1_3_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586182.3616660"},{"key":"e_1_3_3_1_5_2","unstructured":"Leah Asmelash. 2023. These books are being used to train AI. no one told the authors. https:\/\/www.cnn.com\/2023\/10\/08\/style\/ai-books3-authors-nora-roberts-cec"},{"key":"e_1_3_3_1_6_2","doi-asserted-by":"publisher","unstructured":"John\u00a0W Ayers Adam Poliak Mark Dredze Eric\u00a0C Leas Zechariah Zhu Jessica\u00a0B Kelley Dennis\u00a0J Faix Aaron\u00a0M Goodman Christopher\u00a0A Longhurst Michael Hogarth et\u00a0al. 2023. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine 183 6 (2023) 589\u2013596. 10.1001\/jamainternmed.2023.1838","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"e_1_3_3_1_7_2","unstructured":"Scott Barnett Zac Brannelly Stefanus Kurniawan and Sheng Wong. 2024. Fine-tuning or fine-failing? debunking performance myths in large language models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2406.11201 (2024)."},{"key":"e_1_3_3_1_8_2","doi-asserted-by":"crossref","unstructured":"Earl\u00a0T Barr Mark Harman Phil McMinn Muzammil Shahbaz and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41 5 (2014) 507\u2013525.","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"crossref","unstructured":"Manuela Benary Xing\u00a0David Wang Max Schmidt Dominik Soll Georg Hilfenhaus Mani Nassir Christian Sigler Maren Kn\u00f6dler Ulrich Keller Dieter Beule et\u00a0al. 2023. Leveraging large language models for decision support in personalized oncology. JAMA Network Open 6 11 (2023) e2343689\u2013e2343689.","DOI":"10.1001\/jamanetworkopen.2023.43689"},{"key":"e_1_3_3_1_10_2","doi-asserted-by":"crossref","unstructured":"Jessica\u00a0G Borger Ashley\u00a0P Ng Holly Anderton George\u00a0W Ashdown Megan Auld Marnie\u00a0E Blewitt Daniel\u00a0V Brown Melissa\u00a0J Call Peter Collins Saskia Freytag et\u00a0al. 2023. Artificial intelligence takes center stage: exploring the capabilities and implications of ChatGPT and other AI-assisted technologies in scientific research and education. Immunology and cell biology 101 10 (2023) 923\u2013935.","DOI":"10.1111\/imcb.12689"},{"key":"e_1_3_3_1_11_2","doi-asserted-by":"crossref","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3 2 (2006) 77\u2013101.","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_3_1_12_2","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared\u00a0D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et\u00a0al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877\u20131901."},{"key":"e_1_3_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Yangxiao Cai Peng Liang Yifei Wang Zengyang Li and Mojtaba Shahin. 2025. Demystifying issues causes and solutions in LLM open-source projects. Journal of Systems and Software (2025) 112452.","DOI":"10.1016\/j.jss.2025.112452"},{"key":"e_1_3_3_1_14_2","unstructured":"Yixin Cao Shibo Hong Xinze Li Jiahao Ying Yubo Ma Haiyuan Liang Yantao Liu Zijun Yao Xiaozhi Wang Dan Huang et\u00a0al. 2025. Toward generalizable evaluation in the llm era: A survey beyond benchmarks. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2504.18838 (2025)."},{"key":"e_1_3_3_1_15_2","doi-asserted-by":"crossref","unstructured":"Yupeng Chang Xu Wang Jindong Wang Yuan Wu Linyi Yang Kaijie Zhu Hao Chen Xiaoyuan Yi Cunxiang Wang Yidong Wang et\u00a0al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15 3 (2024) 1\u201345.","DOI":"10.1145\/3641289"},{"key":"e_1_3_3_1_16_2","doi-asserted-by":"publisher","unstructured":"Angeline Chatelan Aur\u00e9lien Clerc and Pierre-Alexandre Fonta. 2023. ChatGPT and Future Artificial Intelligence Chatbots: What may be the Influence on Credentialed Nutrition and Dietetics Practitioners?Journal of the Academy of Nutrition and Dietetics 123 (2023) 1525\u20132529. Issue 11. 10.1016\/j.jand.2023.08.001","DOI":"10.1016\/j.jand.2023.08.001"},{"key":"e_1_3_3_1_17_2","unstructured":"Guiming\u00a0Hardy Chen Shunian Chen Ziche Liu Feng Jiang and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2402.10669 (2024)."},{"key":"e_1_3_3_1_18_2","doi-asserted-by":"crossref","unstructured":"Xiang Chen Chaoyang Gao Chunyang Chen Guangbei Zhang and Yong Liu. 2025. An Empirical Study on Challenges for LLM Application Developers. ACM Transactions on Software Engineering and Methodology (2025).","DOI":"10.1145\/3715007"},{"key":"e_1_3_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Szu-Wei Cheng Chung-Wen Chang Wan-Jung Chang Hao-Wei Wang Chih-Sung Liang Taishiro Kishimoto Jane Pei-Chen Chang John\u00a0S Kuo and Kuan-Pin Su. 2023. The now and future of ChatGPT and GPT in psychiatry. Psychiatry and clinical neurosciences 77 11 (2023) 592\u2013596.","DOI":"10.1111\/pcn.13588"},{"key":"e_1_3_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643834.3661515"},{"key":"e_1_3_3_1_21_2","unstructured":"Philippe De\u00a0Wilde Payal Arora Fernando Buarque\u00a0de Lima\u00a0Neto Yik Chin Mamello Thinyane Serge Stinckwich Eleonore Fournier-Tombs and Tshilidzi Marwala. 2024. Recommendations on the use of synthetic data to train ai models. (2024)."},{"key":"e_1_3_3_1_22_2","unstructured":"Fernando Delgado Stephen Yang Michael Madaio and Qian Yang. 2021. Stakeholder Participation in AI: Beyond\" Add Diverse Stakeholders and Stir\". arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2111.01122 (2021)."},{"key":"e_1_3_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i28.35351"},{"key":"e_1_3_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640544.3645216"},{"key":"e_1_3_3_1_25_2","unstructured":"Yann Dubois Bal\u00e1zs Galambosi Percy Liang and Tatsunori\u00a0B Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2404.04475 (2024)."},{"key":"e_1_3_3_1_26_2","unstructured":"Yann Dubois Chen\u00a0Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy\u00a0S Liang and Tatsunori\u00a0B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3729176.3729199"},{"key":"e_1_3_3_1_28_2","volume-title":"The interpretation of cultures","author":"Geertz Clifford","year":"2017","unstructured":"Clifford Geertz. 2017. The interpretation of cultures. Basic books."},{"key":"e_1_3_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Sebastian Gehrmann Elizabeth Clark and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research 77 (2023) 103\u2013166.","DOI":"10.1613\/jair.1.13715"},{"key":"e_1_3_3_1_30_2","volume-title":"Ghost work: How to stop Silicon Valley from building a new global underclass","author":"Gray Mary\u00a0L","year":"2019","unstructured":"Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Harper Business."},{"key":"e_1_3_3_1_31_2","unstructured":"Jiawei Gu Xuhui Jiang Zhichao Shi Hexiang Tan Xuehao Zhai Chengjin Xu Wei Li Yinghan Shen Shengjie Ma Honghao Liu et\u00a0al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2411.15594 (2024)."},{"key":"e_1_3_3_1_32_2","doi-asserted-by":"crossref","unstructured":"Suchin Gururangan Ana Marasovi\u0107 Swabha Swayamdipta Kyle Lo Iz Beltagy Doug Downey and Noah\u00a0A Smith. 2020. Don\u2019t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2004.10964 (2020).","DOI":"10.18653\/v1\/2020.acl-main.740"},{"key":"e_1_3_3_1_33_2","doi-asserted-by":"crossref","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33 8 (2024) 1\u201379.","DOI":"10.1145\/3695988"},{"key":"e_1_3_3_1_34_2","unstructured":"Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng Haotian Wang Qianglong Chen Weihua Peng Xiaocheng Feng Bing Qin et\u00a0al. 2023. A survey on hallucination in large language models: Principles taxonomy challenges and open questions. ACM Transactions on Information Systems (2023)."},{"key":"e_1_3_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v11i1.27547"},{"key":"e_1_3_3_1_36_2","doi-asserted-by":"crossref","unstructured":"Johannes Jakubik Michael V\u00f6ssing Niklas K\u00fchl Jannis Walk and Gerhard Satzger. 2024. Data-centric artificial intelligence. Business & Information Systems Engineering 66 4 (2024) 507\u2013515.","DOI":"10.1007\/s12599-024-00857-8"},{"key":"e_1_3_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACDSA65407.2025.11166110"},{"key":"e_1_3_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Ziwei Ji Nayeon Lee Rita Frieske Tiezheng Yu Dan Su Yan Xu Etsuko Ishii Ye\u00a0Jin Bang Andrea Madotto and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55 12 (2023) 1\u201338.","DOI":"10.1145\/3571730"},{"key":"e_1_3_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642216"},{"key":"e_1_3_3_1_40_2","unstructured":"Xuechen Li Tianyi Zhang Yann Dubois Rohan Taori Ishaan Gulrajani Carlos Guestrin Percy Liang and Tatsunori\u00a0B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https:\/\/github.com\/tatsu-lab\/alpaca_eval."},{"key":"e_1_3_3_1_41_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74\u201381. https:\/\/aclanthology.org\/W04-1013"},{"key":"e_1_3_3_1_42_2","unstructured":"June\u00a0M Liu Donghao Li He Cao Tianhe Ren Zeyi Liao and Jiamin Wu. 2023. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2309.15461 (2023)."},{"key":"e_1_3_3_1_43_2","unstructured":"Yang Liu Yuanshun Yao Jean-Francois Ton Xiaoying Zhang Ruocheng Guo\u00a0Hao Cheng Yegor Klochkov Muhammad\u00a0Faaiz Taufiq and Hang Li. 2023. Trustworthy LLMs: A survey and guideline for evaluating large language models\u2019 alignment. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2308.05374 (2023)."},{"key":"e_1_3_3_1_44_2","doi-asserted-by":"crossref","unstructured":"Adian Liusie Vatsal Raina Yassir Fathullah and Mark Gales. 2024. Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2405.05894 (2024).","DOI":"10.18653\/v1\/2024.emnlp-main.389"},{"key":"e_1_3_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376727"},{"key":"e_1_3_3_1_46_2","volume-title":"Being Ethnographic: A Guide to the Theory and Practice of Ethnography","author":"Madden Raymond","year":"2022","unstructured":"Raymond Madden. 2022. Being Ethnographic: A Guide to the Theory and Practice of Ethnography. SAGE."},{"key":"e_1_3_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-71477-1_5"},{"key":"e_1_3_3_1_48_2","unstructured":"Jiya Manchanda Laura Boettcher Matheus Westphalen and Jasser Jasser. 2024. The open source advantage in large language models (llms). arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2412.12004 (2024)."},{"key":"e_1_3_3_1_49_2","doi-asserted-by":"crossref","unstructured":"Peter\u00a0J Martin. 1999. Influences on clinical judgement in mental health nursing. NT Research 4 4 (1999) 273\u2013281.","DOI":"10.1177\/136140969900400405"},{"key":"e_1_3_3_1_50_2","unstructured":"Benjamin\u00a0Paul Michael. 2019. Clinical Judgement: an investigation of clinical decision-making. Ph.\u00a0D. Dissertation. University of Sheffield."},{"key":"e_1_3_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP66354.2025.00051"},{"key":"e_1_3_3_1_52_2","doi-asserted-by":"crossref","unstructured":"Pauli Pakarinen and Ruthanne Huising. 2025. Relational expertise: What machines can\u2019t know. Journal of Management Studies 62 5 (2025) 2053\u20132082.","DOI":"10.1111\/joms.12915"},{"key":"e_1_3_3_1_53_2","doi-asserted-by":"crossref","unstructured":"Indranil Pan Lachlan\u00a0R Mason and Omar\u00a0K Matar. 2022. Data-centric Engineering: integrating simulation machine learning and statistics. Challenges and opportunities. Chemical Engineering Science 249 (2022) 117271.","DOI":"10.1016\/j.ces.2021.117271"},{"key":"e_1_3_3_1_54_2","doi-asserted-by":"crossref","unstructured":"Qian Pan Zahra Ashktorab Michael Desmond Martin\u00a0Santillan Cooper James Johnson Rahul Nair Elizabeth Daly and Werner Geyer. 2024. Human-Centered Design Recommendations for LLM-as-a-Judge. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2407.03479 (2024).","DOI":"10.18653\/v1\/2024.hucllm-1.2"},{"key":"e_1_3_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Kishore Papineni Salim Roukos Todd Ward and Wei jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. 311\u2013318.","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_3_1_56_2","unstructured":"Chanjun Park and Hyeonwoo Kim. 2024. Understanding llm development through longitudinal study: Insights from the open ko-llm leaderboard. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2409.03257 (2024)."},{"key":"e_1_3_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICoDSE63307.2024.10829889"},{"key":"e_1_3_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2372251.2372255"},{"key":"e_1_3_3_1_59_2","unstructured":"Ji-Lun Peng Sijia Cheng Egil Diau Yung-Yu Shih Po-Heng Chen Yen-Ting Lin and Yun-Nung Chen. 2024. A survey of useful llm evaluation. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2406.00936 (2024)."},{"key":"e_1_3_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3713436"},{"key":"e_1_3_3_1_61_2","unstructured":"Xiangyu Qi Yi Zeng Tinghao Xie Pin-Yu Chen Ruoxi Jia Prateek Mittal and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety even when users do not intend to!arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2310.03693 (2023)."},{"key":"e_1_3_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3714051"},{"key":"e_1_3_3_1_63_2","unstructured":"Shaolun Ruan Rui Sheng Xiaolin Wen Jiachen Wang Tianyi Zhang Yong Wang Tim Dwyer and Jiannan Li. 2025. Qualitative Study for LLM-assisted Design Study Process: Strategies Challenges and Roles. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2507.10024 (2025)."},{"key":"e_1_3_3_1_64_2","first-page":"887","volume-title":"Healthcare","author":"Sallam Malik","year":"2023","unstructured":"Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare , Vol.\u00a011. MDPI, 887."},{"key":"e_1_3_3_1_65_2","unstructured":"Souvika Sarkar Mohammad\u00a0Fakhruddin Babar Monowar Hasan and Shubhra\u00a0Kanti Karmaker. 2024. LLMs as On-demand Customizable Service. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2401.16577 (2024)."},{"key":"e_1_3_3_1_66_2","doi-asserted-by":"publisher","unstructured":"Devansh Saxena Karla Badillo-Urquiola Pamela\u00a0J. Wisniewski and Shion Guha. 2021. A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare. Proc. ACM Hum.-Comput. Interact. 5 CSCW2 Article 348 (Oct. 2021) 41\u00a0pages. 10.1145\/3476089","DOI":"10.1145\/3476089"},{"key":"e_1_3_3_1_67_2","doi-asserted-by":"crossref","unstructured":"Chirag Shah. 2025. From Prompt Engineering to Prompt Science with Humans in the Loop. Commun. ACM 68 6 (2025) 54\u201361.","DOI":"10.1145\/3709599"},{"key":"e_1_3_3_1_68_2","doi-asserted-by":"crossref","unstructured":"Shreya Shankar JD Zamfirescu-Pereira Bj\u00f6rn Hartmann Aditya\u00a0G Parameswaran and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2404.12272 (2024).","DOI":"10.1145\/3654777.3676450"},{"key":"e_1_3_3_1_69_2","unstructured":"Yuchen Shao Yuheng Huang Jiawei Shen Lei Ma Ting Su and Chengcheng Wan. 2024. Are LLMs Correctly Integrated into Software Systems?arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2407.05138 (2024)."},{"key":"e_1_3_3_1_70_2","doi-asserted-by":"crossref","unstructured":"Helen Sharp Yvonne Dittrich and Cleidson\u00a0RB De\u00a0Souza. 2016. The role of ethnographic studies in empirical software engineering. IEEE Transactions on Software Engineering 42 8 (2016) 786\u2013804.","DOI":"10.1109\/TSE.2016.2519887"},{"key":"e_1_3_3_1_71_2","unstructured":"Lin Shi Chiyu Ma Wenhua Liang Xingjian Diao Weicheng Ma and Soroush Vosoughi. 2024. Judging the judges: A systematic study of position bias in llm-as-a-judge. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2406.07791 (2024)."},{"key":"e_1_3_3_1_72_2","unstructured":"Yifan Song Guoyin Wang Sujian Li and Bill\u00a0Yuchen Lin. 2024. The good the bad and the greedy: Evaluation of llms should not ignore non-determinism. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2407.10457 (2024)."},{"key":"e_1_3_3_1_73_2","unstructured":"Chongyan Sun Ken Lin Shiwei Wang Hulong Wu Chengfei Fu and Zhen Wang. 2024. LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2408.13338 (2024)."},{"key":"e_1_3_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3641924"},{"key":"e_1_3_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3708359.3712091"},{"key":"e_1_3_3_1_76_2","unstructured":"Kits\u00a0AI Team. 2025. Friend or faux: The ethics of Ai Voice Training and why it matters. https:\/\/www.kits.ai\/blog\/ai-voice-cloning-ethics"},{"key":"e_1_3_3_1_77_2","doi-asserted-by":"crossref","unstructured":"Shubo Tian Qiao Jin Lana Yeganova Po-Ting Lai Qingqing Zhu Xiuying Chen Yifan Yang Qingyu Chen Won Kim Donald\u00a0C Comeau et\u00a0al. 2023. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings in Bioinformatics 25 1 (2023).","DOI":"10.1093\/bib\/bbad493"},{"key":"e_1_3_3_1_78_2","doi-asserted-by":"crossref","unstructured":"Ruth Vo M Smith and N Patton. 2021. Journal of Human Nutrition and Dietetics 34 1 (2021) 124\u2013133.","DOI":"10.1111\/jhn.12820"},{"key":"e_1_3_3_1_79_2","doi-asserted-by":"crossref","unstructured":"Steven\u00a0Euijong Whang Yuji Roh Hwanjun Song and Jae-Gil Lee. 2023. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal 32 4 (2023) 791\u2013813.","DOI":"10.1007\/s00778-022-00775-9"},{"key":"e_1_3_3_1_80_2","doi-asserted-by":"crossref","unstructured":"Xingjiao Wu Luwei Xiao Yixuan Sun Junhang Zhang Tianlong Ma and Liang He. 2022. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems 135 (2022) 364\u2013381.","DOI":"10.1016\/j.future.2022.05.014"},{"key":"e_1_3_3_1_81_2","doi-asserted-by":"crossref","unstructured":"Lixuan Zhang Katelyn Wilson and Clinton Amos. 2025. The rise of AI art: A look through digital artists\u2019 eyes. First Monday (2025).","DOI":"10.5210\/fm.v30i4.13809"},{"key":"e_1_3_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i17.29934"},{"key":"e_1_3_3_1_83_2","unstructured":"Wayne\u00a0Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et\u00a0al. 2023. A survey of large language models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2303.18223 (2023)."},{"key":"e_1_3_3_1_84_2","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et\u00a0al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_3_1_85_2","unstructured":"Hongyin Zhu. 2024. Challenges and responses in the practice of large language models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2408.09416 (2024)."},{"key":"e_1_3_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3516506"}],"event":{"name":"IUI '26: 31st International Conference on Intelligent User Interfaces","location":"Paphos Cyprus","acronym":"IUI '26","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction","SIGAI ACM Special Interest Group on Artificial Intelligence"]},"container-title":["Proceedings of the 31st International Conference on Intelligent User Interfaces"],"original-title":[],"deposited":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T13:02:18Z","timestamp":1773493338000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3742413.3789105"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,22]]},"references-count":85,"alternative-id":["10.1145\/3742413.3789105","10.1145\/3742413"],"URL":"https:\/\/doi.org\/10.1145\/3742413.3789105","relation":{},"subject":[],"published":{"date-parts":[[2026,3,22]]},"assertion":[{"value":"2026-03-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}