{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T08:01:24Z","timestamp":1780300884151,"version":"3.54.0"},"publisher-location":"New York, NY, USA","reference-count":36,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T00:00:00Z","timestamp":1775952000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,12]]},"DOI":"10.1145\/3786151.3788605","type":"proceedings-article","created":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T07:01:45Z","timestamp":1780297305000},"page":"12-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["In-IDE Toolkit for Developers of AI-Based Features"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7670-5996","authenticated-orcid":false,"given":"Yaroslav","family":"Sokolov","sequence":"first","affiliation":[{"name":"JetBrains, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3142-5703","authenticated-orcid":false,"given":"Yury","family":"Khudyakov","sequence":"additional","affiliation":[{"name":"JetBrains, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0498-3888","authenticated-orcid":false,"given":"Lenar","family":"Sharipov","sequence":"additional","affiliation":[{"name":"JetBrains, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3277-0145","authenticated-orcid":false,"given":"Andrei","family":"Gasparian","sequence":"additional","affiliation":[{"name":"JetBrains, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1621-0653","authenticated-orcid":false,"given":"Parth","family":"Tiwary","sequence":"additional","affiliation":[{"name":"JetBrains, Krakow, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8916-9659","authenticated-orcid":false,"given":"Artem","family":"Trofimov","sequence":"additional","affiliation":[{"name":"JetBrains, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Sangeetha Annam Merry Saxena Ujjwal Kaushik and Shikha Mittal. 2025. LangChain: Simplifying Development with Language Models. Textual Intelligence: Large Language Models and Their Real-World Applications (2025) 287\u2013304.","DOI":"10.1002\/9781394287499.ch12"},{"key":"e_1_3_3_2_3_2","unstructured":"Arize AI Inc.2025. Arize AI. https:\/\/arize.com. AI & Agent Engineering Platform: development observability and evaluation. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_4_2","unstructured":"Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. https:\/\/www.wandb.com\/. Software available from https:\/\/wandb.com. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3658948"},{"key":"e_1_3_3_2_6_2","unstructured":"Alan Chan Kevin Wei Sihao Huang Nitarshan Rajkumar Elija Perrier Seth Lazar Gillian\u00a0K. Hadfield and Markus Anderljung. 2025. Infrastructure for ai agents. https:\/\/arxiv.org\/abs\/2501.10114. arXiv:https:\/\/arXiv.org\/abs\/2501.10114."},{"key":"e_1_3_3_2_7_2","unstructured":"Harrison Chase. 2022. LangChain. https:\/\/github.com\/langchain-ai\/langchain. Open-source Python framework for developing applications powered by language models. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_8_2","unstructured":"Samuel Colvin Eric Jolibois Hasan Ramezani Adrian Garcia\u00a0Badaracco Terrence Dorsey David Montague Serge Matveenko Marcelo Trylesinski Sydney Runkle David Hewitt Alex Hall and Victorien Plot. 2025. Pydantic Validation. https:\/\/docs.pydantic.dev\/latest\/. Version v2.13.0a0+dev. Widely used data validation library for Python 3.9+ with strong typing and JSON Schema support. Source: https:\/\/github.com\/pydantic\/pydantic. License: MIT. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_9_2","unstructured":"Liming Dong Qinghua Lu and Liming Zhu. 2024. AgentOps: Enabling Observability of LLM Agents. https:\/\/arxiv.org\/abs\/2411.05285. arXiv:https:\/\/arXiv.org\/abs\/2411.05285."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Shanghua Gao Ada Fang Yepeng Huang Valentina Giunchiglia Ayush Noori Jonathan\u00a0Richard Schwarz Yasha Ektefaie Jovana Kondic and Marinka Zitnik. 2024. Empowering biomedical discovery with AI agents. Cell 187 22 (2024) 6125\u20136151.","DOI":"10.1016\/j.cell.2024.09.022"},{"key":"e_1_3_3_2_11_2","unstructured":"Gartner Inc.2025. Gartner Identifies the Top Strategic Trends in Software Engineering for 2025 and Beyond. https:\/\/www.gartner.com\/en\/newsroom\/press-releases\/2025-07-01-gartner-identifies-the-top-strategic-trends-in-software-engineering-for-2025-and-beyond. Press release. Contains the prediction that by 2027 at least 55% of software engineering teams will be actively building LLM-based features. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_12_2","unstructured":"Jiawei Gu Jianqing Zhao Chenghao Liu Wenxin Du Zihuiwen Fang Yao Wang Wenqiang Lei and Wayne\u00a0Xin Zhao. 2024. A Survey on LLM-as-a-Judge. https:\/\/arxiv.org\/abs\/2411.15594. arXiv:https:\/\/arXiv.org\/abs\/2411.15594. Survey of LLMs as judges for generative evaluation with techniques to improve reliability and consistency. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_13_2","unstructured":"Jeffrey Ip and Kritin Vongthongsri. 2025. deepeval. https:\/\/github.com\/confident-ai\/deepeval. Version 3.6.8. License: Apache-2.0. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_14_2","unstructured":"JetBrains. 2024. Koog by JetBrains. https:\/\/www.jetbrains.com\/koog\/. Open-source framework for building fault-tolerant scalable production-ready AI agents with a Kotlin DSL; graph-based workflows; built-in tracing\/monitoring; integrates with Weave and Langfuse. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_15_2","unstructured":"JetBrains. 2025. PyCharm: The Only Python IDE You Need. https:\/\/www.jetbrains.com\/pycharm\/. Professional Python IDE with AI-enhanced experience; supports Python databases Jupyter Git Conda and frameworks incl. PyTorch TensorFlow Django Flask FastAPI. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642349"},{"key":"e_1_3_3_2_17_2","unstructured":"LangChain Inc.2024. LangGraph. https:\/\/github.com\/langchain-ai\/langgraph. Low-level orchestration framework and runtime for building long-running stateful agents with durable execution HITL memory and production deployment. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_18_2","unstructured":"LangChain Inc.2024. LangSmith. https:\/\/smith.langchain.com. Tooling for developing debugging evaluating and deploying LLM applications; integrates with LangChain and LangGraph. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_19_2","unstructured":"LangChain Inc.2025. LangChain Callback System. https:\/\/python.langchain.com\/docs\/concepts\/callbacks. Callback system for hooking into LLM app stages; handlers receive start\/end events tokens errors; pass callbacks per-request or at construction. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_20_2","unstructured":"LangChain Inc.2025. LangChain Runnable. https:\/\/python.langchain.com\/api_reference\/core\/runnables\/langchain_core.runnables.base.Runnable.html. Runnable = unit of work that can be invoked batched streamed transformed and composed; LCEL primitives: RunnableSequence RunnableParallel. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_21_2","unstructured":"Microsoft. 2025. AI Toolkit for Visual Studio Code. https:\/\/github.com\/microsoft\/vscode-ai-toolkit. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_22_2","unstructured":"Microsoft. 2025. Visual Studio Code. https:\/\/code.visualstudio.com\/. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3711896.3736555"},{"key":"e_1_3_3_2_24_2","unstructured":"OpenAI. 2025. OpenAI Python SDK. https:\/\/github.com\/openai\/openai-python. Official Python SDK for the OpenAI API with typed sync\/async clients; supports the Responses and Chat Completions APIs vision and more. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_25_2","unstructured":"Pydantic Team. 2025. Pydantic AI. https:\/\/github.com\/pydantic\/pydantic-ai\/. Framework for structured type-safe debuggable AI applications with agentic workflows and function calling. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_26_2","unstructured":"Clemens Rawert Marc Klingen and Maximilian Deichmann. 2023. Langfuse \u2014 Open-Source LLM Engineering Platform. https:\/\/langfuse.com\/. Software available from https:\/\/langfuse.com. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_27_2","unstructured":"Stack Overflow. 2025. Stack Overflow Developer Survey 2025 \u2014 AI agent observability & security. https:\/\/survey.stackoverflow.co\/2025\/ai#ai-agent-observability-and-security. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_28_2","unstructured":"Stack Overflow. 2025. Stack Overflow Developer Survey 2025 \u2014 Technology: Dev IDEs (Professionals that use AI). https:\/\/survey.stackoverflow.co\/2025\/technology#most-popular-technologies-dev-envs-prof-ai. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640794.3665887"},{"key":"e_1_3_3_2_30_2","unstructured":"SuperAnnotate AI Inc.2025. SuperAnnotate. https:\/\/www.superannotate.com. Build efficient human data and evaluation pipelines to ship better agentic multimodal and frontier AI faster. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643796.3648455"},{"key":"e_1_3_3_2_32_2","unstructured":"Weights & Biases. 2024. Weave: Track Test and Improve LLM Applications. https:\/\/wandb.ai\/site\/products\/weave. Track LLM calls in production test prompts and run evaluations for performance and safety. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_33_2","unstructured":"Asaf Yehudai Lilach Eden Alan Li Guy Uziel Yilun Zhao Roy Bar-Haim Arman Cohan and Michal Shmueli-Scheuer. 2025. Survey on evaluation of LLM-based agents. https:\/\/arxiv.org\/abs\/2503.16416. arXiv:https:\/\/arXiv.org\/abs\/2503.16416."},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-naacl.267"},{"key":"e_1_3_3_2_35_2","unstructured":"Matei\u00a0A. Zaharia Andrew Chen Aaron Davidson Ali Ghodsi Sue\u00a0Ann Hong Andy Konwinski Siddharth Murching Tomas Nykodym Paul Ogilvie Mani Parkhe Fen Xie and Corey Zumar. 2018. Accelerating the Machine Learning Lifecycle with MLflow. IEEE Data Engineering Bulletin 41 (2018) 39\u201345. MLflow is an open-source platform for managing the ML lifecycle: tracking packaging registry serving evaluation and observability. Available at https:\/\/mlflow.org. Accessed: 2025-10-27."},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3766882.3767169"},{"key":"e_1_3_3_2_37_2","volume-title":"Forty-second International Conference on Machine Learning","author":"Zhuge Mingchen","year":"2025","unstructured":"Mingchen Zhuge, Changsheng Zhao, Dylan\u00a0R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et\u00a0al. 2025. Agent-as-a-Judge: Evaluate Agents with Agents. In Forty-second International Conference on Machine Learning."}],"event":{"name":"IDE '26: 3rd ACM\/IEEE International Workshop on Integrated Development Environments","location":"Rio de Janeiro Brazil","acronym":"IDE '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS","Faculty of Engineering of University of Porto"]},"container-title":["Proceedings of the 3rd ACM\/IEEE International Workshop on Integrated Development Environments"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786151.3788605","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T07:18:14Z","timestamp":1780298294000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786151.3788605"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,12]]},"references-count":36,"alternative-id":["10.1145\/3786151.3788605","10.1145\/3786151"],"URL":"https:\/\/doi.org\/10.1145\/3786151.3788605","relation":{},"subject":[],"published":{"date-parts":[[2026,4,12]]},"assertion":[{"value":"2026-06-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}