{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T23:14:16Z","timestamp":1789082056442,"version":"build-2803163510"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP220103044"],"award-info":[{"award-number":["DP220103044"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Existing multi-source root cause analysis (RCA) methods for microservice systems assume all services have traces to construct a service call graph. However, this assumption is not practical as microservice systems evolve rapidly and may contain blackbox services without traces, such as compiled software or unsupported services. We refer to these services as blind spots. In the presence of blind spots, the performance of existing multi-source RCA methods may be affected, as they only diagnose visible services on the call graph. To overcome this limitation, we propose TORAI, a novel unsupervised approach that effectively pinpoints fine-grained root causes without relying on the service call graph. Instead, TORAI first measures anomaly severity using available multi-source telemetry data. It then performs clustering to group services based on their severity symptoms and conducts causal analysis to rank services within each severity cluster. Finally, TORAI aggregates the cluster rankings and uses hypothesis testing to identify fine-grained root causes. TORAI provides an unsupervised approach that leverages available multi-source telemetry data for RCA without requiring a constructed service call graph or further intrusive actions, thus addressing the limitations of existing methods. Our experiments on three benchmark systems demonstrate that TORAI outperforms state-of-the-art baselines remarkably in the presence of blind spots. Performance on real-world failures further shows that TORAI can accurately pinpoint the root causes in top-3 recommendations.<\/jats:p>","DOI":"10.1145\/3808137","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"2928-2951","source":"Crossref","is-referenced-by-count":3,"title":["TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7243-3225","authenticated-orcid":false,"given":"Luan","family":"Pham","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2463-7770","authenticated-orcid":false,"given":"Huong","family":"Ha","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5558-3790","authenticated-orcid":false,"given":"Xiuzhen","family":"Zhang","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3063-9425","authenticated-orcid":false,"given":"Hongyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Chongqing University, Chongqing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Container Advisor -an open-source tool to monitor containers. Retrieved Apr 14, 2026 from https:\/\/github.com\/ google\/cadvisor"},{"key":"e_1_2_1_2_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Elasticsearch Log Monitoring -Centralized Log Management. Retrieved Apr 14, 2026 from https:\/\/www.elastic.co\/"},{"key":"e_1_2_1_3_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Gaussian Mixture Model. Retrieved Apr 14, 2026 from https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn. mixture.GaussianMixture.html"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Google -Site Reliability Engineering. Retrieved Apr 14, 2026 from https:\/\/sre.google\/sre-book\/monitoringdistributed-systems\/"},{"key":"e_1_2_1_5_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Grafana Loki -A horizontally scalable, highly available, multi-tenant log aggregation system. Retrieved Apr 14, 2026 from https:\/\/grafana.com\/oss\/loki\/"},{"key":"e_1_2_1_6_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. The Istio service mesh. Retrieved Apr 14, 2026 from https:\/\/istio.io\/"},{"key":"e_1_2_1_7_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Jaeger: open source, distributed tracing platform. Retrieved Apr 14, 2026 from https:\/\/www.jaegertracing.io\/"},{"key":"e_1_2_1_8_1","volume-title":"Retrieved","author":"A","year":"2026","unstructured":"A lightweight, ultra-fast tool for building observability pipelines. Retrieved Apr 14, 2026 from https:\/\/vector.dev\/"},{"key":"e_1_2_1_9_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Online Boutique is a cloud-first microservices demo application. Retrieved Apr 14, 2026 from https:\/\/github. com\/GoogleCloudPlatform\/microservices-demo"},{"key":"e_1_2_1_10_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. An open-source monitoring and alerting toolkit. Retrieved Apr 14, 2026 from https:\/\/prometheus.io\/"},{"key":"e_1_2_1_11_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Sock Shop -A Microservices Demo Application. Retrieved Apr 14, 2026 from https:\/\/github.com\/microservicesdemo\/microservices-demo"},{"key":"e_1_2_1_12_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Stress test for Computer system. Retrieved Apr 14, 2026 from https:\/\/manpages.ubuntu.com\/manpages\/bionic\/ man1\/stress-ng.1.html"},{"key":"e_1_2_1_13_1","volume-title":"Traffic Control. Retrieved","year":"2026","unstructured":"2025. Traffic Control. Retrieved Apr 14, 2026 from https:\/\/man7.org\/linux\/man-pages\/man8\/tc.8.html"},{"key":"e_1_2_1_14_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Train Ticket Benchmark System. Retrieved Apr 14, 2026 from https:\/\/github.com\/FudanSELab\/train-ticket"},{"key":"e_1_2_1_15_1","volume-title":"Retrieved","year":"2026","unstructured":"2025. Uber Microservice Systems. Retrieved Apr 14, 2026 from https:\/\/www.uber.com\/en-AU\/blog\/up-portablemicroservices-ready-for-the-cloud"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672254"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2004.2"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338916"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111906"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3241299"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2017.13"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460345"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556904"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00074"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2259"},{"key":"e_1_2_1_26_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS'20)","author":"Jaber Amin","unstructured":"Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. 2020. Causal Discovery from Soft Interventions with Unknown Targets: Characterization and Learning. In Advances in Neural Information Processing Systems (NeurIPS'20), Vol. 33. 9551-9561."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111793"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00150"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539041"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWQOS52092.2021.9521340"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549092"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.11417"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-14923-8_15"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2022.3174631"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00034"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWQoS49365.2020.9213058"},{"key":"e_1_2_1_38_1","unstructured":"Odigos. 2025. Solving the Pitfalls of Distributed Tracing in Real-World Microservices. https:\/\/odigos.io\/blog\/solvingpitfalls-of-distributed-tracing-in-real-world-microservices. Accessed on September 2 2025."},{"key":"e_1_2_1_39_1","volume-title":"Root Cause Analysis of Outliers with Missing Structural Knowledge. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=7Nxq4RQApu","author":"Orchard William Roy","year":"2025","unstructured":"William Roy Orchard, Nastaran Okati, Sergio Hernan Garrido Mejia, Patrick Bl\u00f6baum, and Dominik Janzing. 2025. Root Cause Analysis of Outliers with Missing Structural Knowledge. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=7Nxq4RQApu"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2020.04.017"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.31925976.v1"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2601"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.31938495.v1"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3660805"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695065"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the ACM on Software Engineering 3, FSE","author":"Pham Luan","year":"2026","unstructured":"Luan Pham, Victor Nicolet, Joey Dodds, Hui Guan, and Daniel Kroening. 2026. EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems. Proceedings of the ACM on Software Engineering 3, FSE (2026)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3715290"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3629526.3645047"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.aau4996"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176344136"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313653"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3603269.3604823"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3501297"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/2074158.2074215"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS49710.2020.00026"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD53861.2021.00072"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","unstructured":"Li Wu. 2022. Automatic performance diagnosis and recovery in cloud microservices. TU Berlin (Germany). doi:10.14279\/ depositonce-14959 10.14279\/depositonce-14959","DOI":"10.14279\/depositonce-14959"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudIntelligence52565.2021.00015"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3734868"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449905"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616249"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_64_1","first-page":"321","volume-title":"20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Zhang Lei","year":"2023","unstructured":"Lei Zhang, Zhiqiang Xie, Vaastav Anand, Ymir Vigfusson, and Jonathan Mace. 2023. The Benefit of Hindsight: Tracing {Edge-Cases} in Distributed Systems. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 321-339."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","unstructured":"Shenglin Zhang Pengxiang Jin Zihan Lin Yongqian Sun Bicheng Zhang Sibo Xia Zhengdan Li Zhenyu Zhong Minghua Ma Wa Jin et al. 2023. Robust failure diagnosis of microservice system through multimodal data. IEEE Transactions on Services Computing (2023). doi:10.1109\/TSC.2023.3290018 10.1109\/TSC.2023.3290018","DOI":"10.1109\/TSC.2023.3290018"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3481903"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589334.3645442"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2887384"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3674726"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808137","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:12:22Z","timestamp":1782843142000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":69,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808137"],"URL":"https:\/\/doi.org\/10.1145\/3808137","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}