{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T17:11:31Z","timestamp":1777655491662,"version":"3.51.4"},"publisher-location":"Cham","reference-count":62,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031731129","type":"print"},{"value":"9783031731136","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T00:00:00Z","timestamp":1732147200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T00:00:00Z","timestamp":1732147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"DOI":"10.1007\/978-3-031-73113-6_10","type":"book-chapter","created":{"date-parts":[[2024,11,20]],"date-time":"2024-11-20T08:47:41Z","timestamp":1732092461000},"page":"161-178","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["OmniACT: A Dataset and\u00a0Benchmark for\u00a0Enabling Multimodal Generalist Autonomous Agents for\u00a0Desktop and\u00a0Web"],"prefix":"10.1007","author":[{"given":"Raghav","family":"Kapoor","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yash Parag","family":"Butala","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Melisa","family":"Russak","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing Yu","family":"Koh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kiran","family":"Kamble","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Waseem","family":"AlShikh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruslan","family":"Salakhutdinov","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,11,21]]},"reference":[{"key":"10_CR1","unstructured":"Pyautogui: a cross-platform GUI automation python module for human beings. https:\/\/github.com\/asweigart\/pyautogui (2023)"},{"key":"10_CR2","unstructured":"AlShikh, W., et al.: Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning (2023)"},{"key":"10_CR3","doi-asserted-by":"crossref","unstructured":"Bai, C., et al.: UIBert: learning generic multimodal representations for UI understanding (2021)","DOI":"10.24963\/ijcai.2021\/235"},{"key":"10_CR4","doi-asserted-by":"crossref","unstructured":"Banerjee, P., Mahajan, S., Arora, K., Baral, C., Riva, O.: Lexi: self-supervised learning of the UI language (2023)","DOI":"10.18653\/v1\/2022.findings-emnlp.519"},{"key":"10_CR5","unstructured":"Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., Plummer, B.A.: Mobile app tasks with iterative feedback (MoTIF): addressing task feasibility in interactive visual environments (2021)"},{"key":"10_CR6","doi-asserted-by":"crossref","unstructured":"Chen, X., et al.: WebSRC: a dataset for web-based structural reading comprehension. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4173\u20134185 (2021)","DOI":"10.18653\/v1\/2021.emnlp-main.343"},{"key":"10_CR7","unstructured":"Chiang, W.L., et al.: Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality (2023). https:\/\/lmsys.org\/blog\/2023-03-30-vicuna\/"},{"key":"10_CR8","doi-asserted-by":"publisher","unstructured":"Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845\u2013854. UIST \u201917, Association for Computing Machinery, New York, NY, USA (2017). https:\/\/doi.org\/10.1145\/3126594.3126651","DOI":"10.1145\/3126594.3126651"},{"key":"10_CR9","unstructured":"Deng, X., et al.: Mind2Web: towards a generalist agent for the web. arXiv preprint arXiv:2306.06070 (2023)"},{"key":"10_CR10","unstructured":"Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: efficient finetuning of quantized LLMs (2023)"},{"key":"10_CR11","unstructured":"Furuta, H., Nachum, O., Lee, K.H., Matsuo, Y., Gu, S.S., Gur, I.: Multimodal web navigation with instruction-finetuned foundation models. arXiv preprint arXiv:2305.11854 (2023)"},{"key":"10_CR12","doi-asserted-by":"crossref","unstructured":"Gupta, T., Kembhavi, A.: Visual programming: compositional visual reasoning without training. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953\u201314962 (2023)","DOI":"10.1109\/CVPR52729.2023.01436"},{"key":"10_CR13","unstructured":"Gur, I., et al.: A real-world WebAgent with planning, long context understanding, and program synthesis (2024)"},{"key":"10_CR14","unstructured":"Gur, I., Rueckert, U., Faust, A., Hakkani-Tur, D.: Learning to navigate the web. In: International Conference on Learning Representations (2018)"},{"key":"10_CR15","doi-asserted-by":"crossref","unstructured":"He, Z., et al.: ActionBert: leveraging user actions for semantic understanding of user interfaces (2021)","DOI":"10.1609\/aaai.v35i7.16741"},{"key":"10_CR16","unstructured":"Huang, Z., Zeng, Z., Liu, B., Fu, D., Fu, J.: Pixel-BERT: aligning image pixels with text by deep multi-modal transformers (2020)"},{"key":"10_CR17","unstructured":"Humphreys, P.C., et al.: A data-driven approach for learning to control computers. In: International Conference on Machine Learning, pp. 9466\u20139482. PMLR (2022)"},{"key":"10_CR18","unstructured":"Kim, G., Baldi, P., McAleer, S.: Language models can solve computer tasks. arXiv preprint arXiv:2303.17491 (2023)"},{"key":"10_CR19","doi-asserted-by":"crossref","unstructured":"Kirillov, A., et al.: Segment anything (2023)","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"10_CR20","doi-asserted-by":"crossref","unstructured":"Koh, J.Y., et al.: VisualWebArena: evaluating multimodal agents on realistic visual web tasks (2024)","DOI":"10.18653\/v1\/2024.acl-long.50"},{"key":"10_CR21","unstructured":"LeCun, Y.: A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27 (2022)"},{"key":"10_CR22","unstructured":"Li, G., Li, Y.: Spotlight: mobile UI understanding using vision-language models with a focus (2023)"},{"key":"10_CR23","doi-asserted-by":"crossref","unstructured":"Li, Y., He, J., Zhou, X., Zhang, Y., Baldridge, J.: Mapping natural language instructions to mobile UI action sequences (2020)","DOI":"10.18653\/v1\/2020.acl-main.729"},{"key":"10_CR24","doi-asserted-by":"crossref","unstructured":"Li, Y., Li, G., He, L., Zheng, J., Li, H., Guan, Z.: Widget captioning: generating natural language description for mobile user interface elements (2020)","DOI":"10.18653\/v1\/2020.emnlp-main.443"},{"key":"10_CR25","unstructured":"Li, Y., Li, G., Zhou, X., Dehghani, M., Gritsenko, A.: VUT: versatile UI transformer for multi-modal multi-task user interface modeling (2021)"},{"key":"10_CR26","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)","DOI":"10.1109\/CVPR52733.2024.02484"},{"key":"10_CR27","unstructured":"Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023)"},{"key":"10_CR28","unstructured":"Liu, X., et al.: AgentBench: evaluating LLMs as agents (2023)"},{"key":"10_CR29","unstructured":"Lu, P., et al.: Chameleon: plug-and-play compositional reasoning with large language models (2023)"},{"key":"10_CR30","unstructured":"Lyu, C., et al.: Macaw-LLM: multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093 (2023)"},{"key":"10_CR31","unstructured":"Nakano, R., et\u00a0al.: WebGPT: browser-assisted question-answering with human feedback (2021)"},{"key":"10_CR32","unstructured":"OpenAI: GPT-4 technical report (2023)"},{"key":"10_CR33","unstructured":"OpenAI: Introducing ChatGPT (2023). https:\/\/openai.com\/blog\/chatgpt"},{"key":"10_CR34","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311\u2013318 (2002)","DOI":"10.3115\/1073083.1073135"},{"key":"10_CR35","unstructured":"Rawles, C., Li, A., Rodriguez, D., Riva, O., Lillicrap, T.: Android in the wild: a large-scale dataset for android device control (2023)"},{"key":"10_CR36","unstructured":"Ren, S., et al.: CodeBLEU: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)"},{"key":"10_CR37","unstructured":"Rozi\u00e8re, B., et al.: Code Llama: open foundation models for code (2023)"},{"key":"10_CR38","unstructured":"Rozi\u00e8re, B., et al.: Code Llama: open foundation models for code (2023)"},{"key":"10_CR39","unstructured":"Shaw, P., et al.: From pixels to UI actions: learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00245 (2023)"},{"key":"10_CR40","unstructured":"Shi, T., Karpathy, A., Fan, L., Hernandez, J., Liang, P.: World of bits: an open-domain platform for web-based agents. In: International Conference on Machine Learning, pp. 3135\u20133144. PMLR (2017)"},{"key":"10_CR41","unstructured":"Sridhar, A., Lo, R., Xu, F.F., Zhu, H., Zhou, S.: Hierarchical prompting assists large language model on web navigation. arXiv preprint arXiv:2305.14257 (2023)"},{"key":"10_CR42","doi-asserted-by":"crossref","unstructured":"Sun, L., Chen, X., Chen, L., Dai, T., Zhu, Z., Yu, K.: Meta-GUI: towards multi-modal conversational agents on mobile GUI. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6699\u20136712 (2022)","DOI":"10.18653\/v1\/2022.emnlp-main.449"},{"key":"10_CR43","doi-asserted-by":"crossref","unstructured":"Sur\u00eds, D., Menon, S., Vondrick, C.: ViperGPT: visual inference via python execution for reasoning (2023)","DOI":"10.1109\/ICCV51070.2023.01092"},{"key":"10_CR44","unstructured":"Team, G., et\u00a0al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)"},{"key":"10_CR45","unstructured":"Team, W.E.: InstructPalmyra-30b: instruct tuned Palmyra-Large model. https:\/\/dev.writer.com (2023)"},{"key":"10_CR46","unstructured":"Team, W.E.: Palmyra-base parameter autoregressive language model. https:\/\/dev.writer.com (2023)"},{"key":"10_CR47","unstructured":"Touvron, H., et al.: LLaMA: open and efficient foundation language models (2023)"},{"key":"10_CR48","doi-asserted-by":"crossref","unstructured":"Wang, B., Li, G., Zhou, X., Chen, Z., Grossman, T., Li, Y.: Screen2Words: automatic mobile UI summarization with multimodal learning. In: The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 498\u2013510 (2021)","DOI":"10.1145\/3472749.3474765"},{"key":"10_CR49","unstructured":"Wang, L., et\u00a0al.: A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 (2023)"},{"key":"10_CR50","first-page":"5776","volume":"33","author":"W Wang","year":"2020","unstructured":"Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Adv. Neural. Inf. Process. Syst. 33, 5776\u20135788 (2020)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"10_CR51","doi-asserted-by":"crossref","unstructured":"Wang, X., et al.: Large-scale multi-modal pre-trained models: a comprehensive survey (2023)","DOI":"10.1007\/s11633-022-1410-8"},{"issue":"4","key":"10_CR52","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","volume":"13","author":"Z Wang","year":"2004","unstructured":"Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600\u2013612 (2004)","journal-title":"IEEE Trans. Image Process."},{"key":"10_CR53","doi-asserted-by":"crossref","unstructured":"Xu, N., Masling, S., Du, M., Campagna, G., Heck, L., Landay, J., Lam, M.: Grounding open-domain instructions to automate web support tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1022\u20131032 (2021)","DOI":"10.18653\/v1\/2021.naacl-main.80"},{"key":"10_CR54","unstructured":"Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4v (2023)"},{"key":"10_CR55","unstructured":"Yang, Z., et al.: The dawn of LMMs: preliminary explorations with GPT-4v (ISION). arXiv preprint arXiv:2309.174219 (2023)"},{"key":"10_CR56","first-page":"20744","volume":"35","author":"S Yao","year":"2022","unstructured":"Yao, S., Chen, H., Yang, J., Narasimhan, K.: WebShop: towards scalable real-world web interaction with grounded language agents. Adv. Neural. Inf. Process. Syst. 35, 20744\u201320757 (2022)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"10_CR57","doi-asserted-by":"crossref","unstructured":"Yin, S., et al.: A survey on multimodal large language models (2023)","DOI":"10.1093\/nsr\/nwae403"},{"key":"10_CR58","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT (2020)"},{"key":"10_CR59","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhang, A.: You only look at screens: multimodal chain-of-action agents (2023)","DOI":"10.18653\/v1\/2024.findings-acl.186"},{"key":"10_CR60","unstructured":"Zhao, W.X., et\u00a0al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)"},{"key":"10_CR61","doi-asserted-by":"crossref","unstructured":"Zhou, S., Alon, U., Agarwal, S., Neubig, G.: CodeBERTScore: evaluating code generation with pretrained models of code. arXiv preprint arXiv:2302.05527 (2023)","DOI":"10.18653\/v1\/2023.emnlp-main.859"},{"key":"10_CR62","unstructured":"Zhou, S., et\u00a0al.: WebArena: a realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 (2023)"}],"container-title":["Lecture Notes in Computer Science","Computer Vision \u2013 ECCV 2024"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-73113-6_10","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,20]],"date-time":"2024-11-20T10:06:34Z","timestamp":1732097194000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-73113-6_10"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,21]]},"ISBN":["9783031731129","9783031731136"],"references-count":62,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-73113-6_10","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,21]]},"assertion":[{"value":"21 November 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"As a part of the dataset creation process, we carefully review the pipeline at every stage, ensuring there is no personally identifiable information or offensive content, either during data collection or through the use of LLMs. For all purposes, we create dummy accounts that mimic real-like user content. To get the gold labels scripts we seek help from well-qualified student workers, approved through the institution, and get the bounding box data annotated through MTurk workers, both of whom are paid  per hour, which is greater than the minimum wage rate (We detail this process in the supplementary material). Human studies are also done with the help of student workers approved by the institution at the above-mentioned payscale. We also ensure that all groups have equitable representation and that no personal opinions are reflected in the dataset, avoiding bias during the collection as well as the annotation process.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Statement"}},{"value":"ECCV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"European Conference on Computer Vision","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Milan","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"29 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 October 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"18","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"eccv2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/eccv2024.ecva.net\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}