{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:17:04Z","timestamp":1750220224241,"version":"3.41.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,2,1]],"date-time":"2022-02-01T00:00:00Z","timestamp":1643673600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2022,8,31]]},"abstract":"<jats:p>\n            Cloud-based Natural Language Understanding (NLU) services are becoming more popular with the development of artificial intelligence. More applications are integrated with cloud-based NLU services to enhance the way people communicate with machines. However, with NLU services provided by different companies powered by unrevealed AI technology, how to choose the best one is a problem for developers. Existing tools that can provide guidance to developers and make recommendations based on their needs are severely limited. This article comprehensively evaluates multiple state-of-the-art NLU services, and the results indicate that there is no absolute winner for different usage requirements. Motivated by this observation, we provide several insights and propose\u00a0\n            <jats:italic>NLUBroker<\/jats:italic>\n            , a Quality of Experience-driven (QoE-driven) broker system, to select the proper service according to the environment.\n            <jats:italic>NLUBroker<\/jats:italic>\n            senses the client and service status and leverages a solution to the multi-armed bandit problem to conduct online learning, aiming to achieve maximum expected QoE. The performance of\u00a0\n            <jats:italic>NLUBroker<\/jats:italic>\n            is evaluated in both simulation and real-world environments, and the evaluation results demonstrate that\u00a0\n            <jats:italic>NLUBroker<\/jats:italic>\n            is an efficient solution for selecting NLU services. It is adaptive to changes in the environment, outperforms three baseline methods we evaluated and improves overall QoE up to 1.5\u00d7 for the evaluated state-of-the-art NLU services.\n          <\/jats:p>","DOI":"10.1145\/3497807","type":"journal-article","created":{"date-parts":[[2022,2,1]],"date-time":"2022-02-01T17:37:06Z","timestamp":1643737026000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["<i>NLUBroker<\/i>\n            : A QoE-driven Broker System for Natural Language Understanding Services"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3442-8411","authenticated-orcid":false,"given":"Lanyu","family":"Xu","sequence":"first","affiliation":[{"name":"Oakland University, Rochester, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arun","family":"Iyengar","sequence":"additional","affiliation":[{"name":"Intelligent Data Management and Analytics, LLC, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weisong","family":"Shi","sequence":"additional","affiliation":[{"name":"Wayne State University, Detroit, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,2]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Amazon. 2020. Amazon Lex. Retrieved from https:\/\/aws.amazon.com\/lex\/."},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3229710.3229742"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944941"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2017.04.046"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5522"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_3_3_8_2","article-title":"Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces","author":"Coucke Alice","year":"2018","unstructured":"Alice Coucke, Alaa Saade, Adrien Ball, Th\u00e9odore Bluche, Alexandre Caulier, David Leroy, Cl\u00e9ment Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv:1805.10190, Retrieved from https:\/\/arxiv.org\/abs\/1805.10190.","journal-title":"arXiv:1805.10190"},{"key":"e_1_3_3_9_2","doi-asserted-by":"crossref","unstructured":"Amir Vahid Dastjerdi and Rajkumar Buyya. 2017. A taxonomy of QoS management and service selection methodologies for cloud computing. In Cloud Computing . CRC Press 109\u2013131.","DOI":"10.1201\/b11149-6"},{"key":"e_1_3_3_10_2","unstructured":"Cem Dilmegani. 2021. Guide to Choose Your Chatbot Platform: Top 5 Systems Reviewed. Retrieved January 6th 2022 from https:\/\/research.aimultiple.com\/dialogflow\/."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2016.59"},{"key":"e_1_3_3_12_2","volume-title":"AI as a Service: Serverless Machine Learning with AWS","author":"Elger Peter","year":"2020","unstructured":"Peter Elger and E\u00f3in Shanaghy. 2020. AI as a Service: Serverless Machine Learning with AWS. Manning Publications."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274657"},{"key":"e_1_3_3_14_2","unstructured":"Facebook. 2020. Wit.ai. Retrieved from https:\/\/wit.ai\/."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304013"},{"key":"e_1_3_3_16_2","unstructured":"Google. 2019. Dialogflow. Retrieved from https:\/\/cloud.google.com\/dialogflow."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.5555\/530588"},{"key":"e_1_3_3_18_2","first-page":"409","article-title":"A broker-based framework for the recommendation of cloud services: A research proposal","volume":"12066","author":"Hentschel Raoul","year":"2020","unstructured":"Raoul Hentschel and Susanne Strahringer. 2020. A broker-based framework for the recommendation of cloud services: A research proposal. Respons. Des. Implement. Use Inf Commun. Technol. 12066 (2020), 409.","journal-title":"Respons. Des. Implement. Use Inf Commun. Technol."},{"key":"e_1_3_3_19_2","volume-title":"ZeroMQ: Messaging for Many Applications","author":"Hintjens Pieter","year":"2013","unstructured":"Pieter Hintjens. 2013. ZeroMQ: Messaging for Many Applications. O\u2019Reilly Media, Inc."},{"key":"e_1_3_3_20_2","unstructured":"IBM. [n.d.]. IBM Watson Products and Solutions. Retrieved from https:\/\/www.ibm.com\/watson\/services\/conversation\/."},{"key":"e_1_3_3_21_2","article-title":"The top 10 chatbots for enterprise customer service","author":"Jacobs Ian","year":"2017","unstructured":"Ian Jacobs, Stephen Powers, Bill Seguin, and Diane Lynch. 2017. The top 10 chatbots for enterprise customer service. Forrest. Rep. (2017).","journal-title":"Forrest. Rep."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10586-019-02928-y"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-020-01747-z"},{"key":"e_1_3_3_24_2","unstructured":"Atakan Kantarci. 2021. Natural Language Platforms: Top NLP APIs & Comparison. Retrieved January 6th 2022 from https:\/\/research.aimultiple.com\/natural-language-platforms\/."},{"key":"e_1_3_3_25_2","unstructured":"Scikit learn. 2020. Scikit-Learn Machine Learning in Python. Retrieved from https:\/\/scikit-learn.org\/."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/1879141.1879143"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772758"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","unstructured":"Xingkun Liu Arash Eshghi Pawel Swietojanski and Verena Rieser. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction . Springer 165\u2013183.","DOI":"10.1007\/978-981-15-9323-9_15"},{"key":"e_1_3_3_29_2","unstructured":"Microsoft. 2019. Language Understanding (LUIS). Retrieved from https:\/\/www.luis.ai\/."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.5555\/2602265"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.06.010"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2019.8737560"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13174-015-0042-4"},{"key":"e_1_3_3_34_2","first-page":"271","article-title":"Evaluation of NLP systems","volume":"57","author":"Resnik Philip","year":"2010","unstructured":"Philip Resnik and Jimmy Lin. 2010. Evaluation of NLP systems. Handbk. Comput. Ling. Nat. Lang. Process. 57 (2010), 271\u2013295.","journal-title":"Handbk. Comput. Ling. Nat. Lang. Process."},{"key":"e_1_3_3_35_2","doi-asserted-by":"crossref","unstructured":"Kumar Shridhar Amit Sahu Ayushman Dash Pedro Alonso Gustav Pihlgren Vinay Pondeknath Fotini Simistira and Marcus Liwicki. 2019. Subword semantic hashing for intent classification on small datasets. In International Joint Conference on Neural Networks (IJCNN\u201919) . IEEE 1\u20136.","DOI":"10.1109\/IJCNN.2019.8852420"},{"key":"e_1_3_3_36_2","unstructured":"Team SimPy. 2020. Simpy: Discrete Event Simulation for Python. Retrieved from https:\/\/simpy.readthedocs.io\/en\/latest\/."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2014.07.019"},{"key":"e_1_3_3_38_2","unstructured":"Tendrl. 2017. Best Practices for Response Times and Latency. Retrieved from https:\/\/github.com\/Tendrl\/documentation\/wiki\/Best-Practices-for-Response-Times-and-Latency."},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2014.6835958"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2991561.2991562"},{"key":"e_1_3_3_41_2","first-page":"2392","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Wu Weiqiang","year":"2020","unstructured":"Weiqiang Wu, Jing Yang, and Cong Shen. 2020. Stochastic linear contextual bandits with diverse contexts. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR, 2392\u20132401."},{"key":"e_1_3_3_42_2","volume-title":"Proceedings of the 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud\u201919)","author":"Xu Lanyu","year":"2019","unstructured":"Lanyu Xu, Arun Iyengar, and Weisong Shi. 2019. NLUBroker: A flexible and responsive broker for cloud-based natural language understanding services. In Proceedings of the 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud\u201919)."}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3497807","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3497807","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:04Z","timestamp":1750188604000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3497807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2]]},"references-count":41,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,8,31]]}},"alternative-id":["10.1145\/3497807"],"URL":"https:\/\/doi.org\/10.1145\/3497807","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"type":"print","value":"1533-5399"},{"type":"electronic","value":"1557-6051"}],"subject":[],"published":{"date-parts":[[2022,2]]},"assertion":[{"value":"2021-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}