{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T21:56:22Z","timestamp":1782770182556,"version":"3.54.5"},"reference-count":25,"publisher":"American Association for the Advancement of Science (AAAS)","content-domain":{"domain":["spj.science.org"],"crossmark-restriction":true},"short-container-title":["Intell Comput"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:p>In recent years, artificial intelligence (AI) has made incredible progress. Advanced foundation models such as ChatGPT can offer powerful conversation, in-context learning, and code generation abilities for a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on their acquired common-sense knowledge. Nonetheless, they still face difficulties in specialized tasks because they lack sufficient domain-specific data during pretraining and can make errors in neural network computations requiring accurate execution. However, many existing models and systems can perform domain-specific tasks very well, although they are not easily accessible or compatible with foundation models because of the different implementations or working mechanisms. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match the subtasks in the outlines to off-the-shelf models and systems with special functionalities to complete these subtasks. Inspired by this, we introduce TaskMatrix.AI as a new AI ecosystem that connects foundation models to millions of application programming interfaces (APIs) for task completion. Unlike most previous studies, which aimed to improve a single AI model, TaskMatrix.AI focuses on using an existing foundation model (as a brain-like central system) and APIs of other AI models and systems (as subtask solvers) to realize diversified tasks in both the digital and physical domains.<\/jats:p>","DOI":"10.34133\/icomputing.0063","type":"journal-article","created":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T16:26:31Z","timestamp":1699892791000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark_01","source":"Crossref","is-referenced-by-count":83,"title":["TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs"],"prefix":"10.34133","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6595-5145","authenticated-orcid":true,"given":"Yaobo","family":"Liang","sequence":"first","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenfei","family":"Wu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting","family":"Song","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenshan","family":"Wu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Xia","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu","family":"Liu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Ou","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuai","family":"Lu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Ji","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaoguang","family":"Mao","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun","family":"Wang","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Linjun","family":"Shou","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Gong","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nan","family":"Duan","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","published-online":{"date-parts":[[2024,2,16]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Devlin J Chang MW Lee K Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv. 2018. https:\/\/doi.org\/10.48550\/arXiv.1810.04805"},{"key":"e_1_3_3_3_2","unstructured":"Dosovitskiy A Beyer L Kolesnikov A Weissenborn D Zhai X Unterthiner T Dehghani M Minderer M Heigold G Gelly S. et\u00a0al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Paper presented at: 9th International Conference on Learning Representations ICLR 2021; 2021 May 3\u20137; Virtual Event Austria."},{"key":"e_1_3_3_4_2","unstructured":"Radford A Kim JW Xu T Brockman G McLeavey C Sutskever I. Robust speech recognition via large-scale weak supervision. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2212.04356"},{"key":"e_1_3_3_5_2","unstructured":"OpenAI. GPT-4 technical report. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.08774"},{"key":"e_1_3_3_6_2","unstructured":"Brown TB Mann B Ryder N Subbiah M Kaplan J Dhariwal P Neelakantan A Shyam P Sastry G Askell A et\u00a0al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;1877\u20131901."},{"key":"e_1_3_3_7_2","unstructured":"Chen M Tworek J Jun H Yuan Q de Oliveira Pinto HP Kaplan J Edwards H Burda Y Joseph N Brockman G et\u00a0al. Evaluating large language models trained on code. arXiv. 2021. https:\/\/doi.org\/10.48550\/arXiv.2107.03374"},{"key":"e_1_3_3_8_2","unstructured":"Ramesh A Pavlov M Goh G Gray S Voss C Radford A Chen M Sutskever I et\u00a0al. Zero-shot text-to-image generation. Paper presented at: International Conference on Machine Learning. Proceedings of Machine Learning Research; July 2021. Virtual event. https:\/\/proceedings.mlr.press\/v139\/ramesh21a.html"},{"key":"e_1_3_3_9_2","unstructured":"Nakano R Hilton J Balaji S Wu J Ouyang L Kim C Hesse C Jain S Kosaraju V Saunders W et\u00a0al. Webgpt: Browser-assisted question-answering with human feedback. arXiv. 2021. https:\/\/doi.org\/10.48550\/arXiv.2112.09332"},{"key":"e_1_3_3_10_2","unstructured":"Yao S Zhao J Yu D Du N Shafran I Narasimhan K Cao Y et\u00a0al. ReAct: Synergizing reasoning and acting in language models. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2210.03629"},{"key":"e_1_3_3_11_2","unstructured":"Lazaridou A Gribovskaya E Stokowiec W Grigorev N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2203.05115"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","unstructured":"Vemprala S Bonatti R Bucker A and Kapoor A. ChatGPT for Robotics: Design Principles and Model Abilities. Tech. rep. MSR-TR-2023-8. Microsoft 2023. https:\/\/www.microsoft.com\/en- us\/research\/publication\/chatgpt- for- robotics- design- principles- and- model-abilities\/","DOI":"10.1109\/ACCESS.2024.3387941"},{"key":"e_1_3_3_13_2","unstructured":"Ahn M Brohan A Brown N Chebotar Y Cortes O David B Finn C Fu C Gopalakrishnan K Hausman K et\u00a0al. Do as I can not as I say: Grounding language in robotic affordances. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2204.01691"},{"key":"e_1_3_3_14_2","unstructured":"Driess D Xia F Sajjadi MSM Lynch C Chowdhery A Ichter B Wahid A Tompson J Vuong Q Yu T et\u00a0al. PaLM-E: An embodied multimodal language model. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.03378"},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","unstructured":"Liang J Huang W Xia F Xu P Hausman K Ichter B Florence P Zeng A et\u00a0al. Code as policies: Language model programs for embodied control. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2209.07753","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"e_1_3_3_16_2","unstructured":"Cobbe K Kosaraju V Bavarian M Chen M Jun H Kaiser L Plappert M Tworek J Hilton J Nakano R et\u00a0al. Training verifiers to solve math word problems. arXiv. 2021. https:\/\/doi.org\/10.48550\/arXiv.2110.14168"},{"key":"e_1_3_3_17_2","unstructured":"Gao L Madaan A Zhou S Alon U Liu P Yang Y Callan J Neubig G et\u00a0al. PAL: Program-aided language models. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2211.10435"},{"key":"e_1_3_3_18_2","unstructured":"Jiang AQ Welleck S Zhou JP Li W Liu J Jamnik M Lacroix T Wu Y Lample G et\u00a0al. Draft sketch and prove: Guiding formal theorem provers with informal proofs. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2210.12283"},{"key":"e_1_3_3_19_2","unstructured":"Schick T Dwivedi-Yu J Dess\u00ec R Raileanu R Lomeli M Zettlemoyer L Cancedda N Scialom T et\u00a0al. Toolformer: Language models can teach themselves to use tools. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2302.04761"},{"key":"e_1_3_3_20_2","unstructured":"Paranjape B Lundberg S Singh S Hajishirzi H Zettlemoyer L Ribeiro MT. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.09014"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Ghazal A Rabl T Hu M Raab F Poess M Crolotte A Jacobsen H-A et\u00a0al. Bigbench: Towards an industry standard benchmark for big data analytics. Paper presented at: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. June 2013; New York NY.","DOI":"10.1145\/2463676.2463712"},{"key":"e_1_3_3_22_2","unstructured":"Mialon G Dess\u00ec R Lomeli M Nalmpantis C Pasunuru R Raileanu R Rozi\u00e8re B Schick T Dwivedi-Yu J Celikyilmaz A et\u00a0al. Augmented language models: A survey. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2302.07842"},{"key":"e_1_3_3_23_2","unstructured":"Wu C Yin S Qi W Wang X Tang Z Duan N. Visual ChatGPT: Talking drawing and editing with visual foundation models. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.04671"},{"key":"e_1_3_3_24_2","unstructured":"Yang Z Li L Wang J Lin K Azarnasab E Ahmed F Liu Z Liu C Zeng M Wang L et\u00a0al. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. arXiv. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.11381"},{"key":"e_1_3_3_25_2","unstructured":"Ouyang L Wu J Jiang X Almeida D Wainwright CL Mishkin P Zhang C Agarwal S Slama K Ray A et\u00a0al. Training language models to follow instructions with human feedback. arXiv. 2022. https:\/\/doi.org\/10.48550\/arXiv.2203.02155"},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","unstructured":"Wu J Zhang X Nichols J Bigham JP. Screen parsing: Towards reverse engineering of UI models from screenshots. Paper presented at: The 34th Annual ACM Symposium on User Interface Software and Technology. October 2021. Virtual event. https:\/\/dl.acm.org\/doi\/fullHtml\/10.1145\/3472749.3474763","DOI":"10.1145\/3472749.3474763"}],"container-title":["Intelligent Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/spj.science.org\/doi\/pdf\/10.34133\/icomputing.0063","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T22:39:57Z","timestamp":1730500797000},"score":1,"resource":{"primary":{"URL":"https:\/\/spj.science.org\/doi\/10.34133\/icomputing.0063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1]]},"references-count":25,"alternative-id":["10.34133\/icomputing.0063"],"URL":"https:\/\/doi.org\/10.34133\/icomputing.0063","relation":{},"ISSN":["2771-5892"],"issn-type":[{"value":"2771-5892","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1]]},"assertion":[{"value":"2023-08-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-06","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"0063"}}