{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T15:17:54Z","timestamp":1776784674196,"version":"3.51.2"},"reference-count":14,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2010,3,12]],"date-time":"2010-03-12T00:00:00Z","timestamp":1268352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2010,3,12]]},"abstract":"<jats:p>Dependencies among system components are crucial to locating root errors in a distributed system. In this paper, we propose an approach to mine intercomponent dependencies from unstructured logs. The technique requires neither additional system instrumentation nor any application specific knowledge. In the approach, we first parse each log message into its log key and parameters. Then, we find dependent log key pairs belong to different components by leveraging co-occurrence analysis and parameter correspondence. After that, we use Bayesian decision theory to estimate the dependency direction of each dependent log key pair. We further apply time delay consistency to remove false positive detections. Case studies on Hadoop show that the technique successfully identifies the dependencies among the distributed system components.<\/jats:p>","DOI":"10.1145\/1740390.1740411","type":"journal-article","created":{"date-parts":[[2010,3,19]],"date-time":"2010-03-19T19:22:47Z","timestamp":1269026567000},"page":"91-96","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":67,"title":["Mining dependency in distributed systems through unstructured logs analysis"],"prefix":"10.1145","volume":"44","author":[{"given":"Jian-Guang","family":"Lou","sequence":"first","affiliation":[{"name":"Microsoft Research Asia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiang","family":"Fu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yi","family":"Wang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiang","family":"Li","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2010,3,12]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proc. of 5th Workshop on Hot Topics in Networks","author":"Bahl P.","year":"2006","unstructured":"P. Bahl , P. Barham , R. Black , R. Chandra , M. Goldszmidt , R. Isaacs , S. Kandula , L. Li , J. MacCormick , D.A. Maltz , R. Mortier , M. Wawrzoniak , and M. Zhang , \" Discovering Dependencies for Network Management \", in Proc. of 5th Workshop on Hot Topics in Networks , 2006 . P. Bahl, P. Barham, R. Black, R. Chandra, M. Goldszmidt, R. Isaacs, S. Kandula, L. Li, J. MacCormick, D.A. Maltz, R. Mortier, M. Wawrzoniak, and M. Zhang, \"Discovering Dependencies for Network Management\", in Proc. of 5th Workshop on Hot Topics in Networks, 2006."},{"key":"e_1_2_1_2_1","volume-title":"USENIX OSDI'08","author":"Chen X.","year":"2008","unstructured":"X. Chen , M. Zhang , Z. M. Mao , and P. Bahl , \" Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions \", USENIX OSDI'08 , 2008 . X. Chen, M. Zhang, Z. M. Mao, and P. Bahl, \"Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions\", USENIX OSDI'08, 2008."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-39671-0_24"},{"key":"e_1_2_1_4_1","volume-title":"TechReport, MSR-TR-2008-67","author":"Barham P.","year":"2008","unstructured":"P. Barham , R. Black , M. Goldszmidt , R. Isaacs , J. MacCormick , R. Mortier , and A. Simma , \" Constellation: automated discovery of service and host dependencies in networked systems \", TechReport, MSR-TR-2008-67 , April , 2008 . P. Barham, R. Black, M. Goldszmidt, R. Isaacs, J. MacCormick, R. Mortier, and A. Simma, \"Constellation: automated discovery of service and host dependencies in networked systems\", TechReport, MSR-TR-2008-67, April, 2008."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1402958.1402970"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1282380.1282383"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1177080.1177096"},{"key":"e_1_2_1_8_1","first-page":"171","volume-title":"Proc. of 15th Int. Workshop on Distributed Systems: Operations and Management (DSOM'04)","author":"Agarwal M.","year":"2004","unstructured":"M. Agarwal , K. Appleby , M. Gupta , G. Kar , A. Neogi , A. Sailer , \" Problem Determination Using Dependency Graphs and Run-time Behavior Models\" , in Proc. of 15th Int. Workshop on Distributed Systems: Operations and Management (DSOM'04) , pp. 171 -- 182 , Nov. 2004 . M. Agarwal, K. Appleby, M. Gupta, G. Kar, A. Neogi, A. Sailer, \"Problem Determination Using Dependency Graphs and Run-time Behavior Models\", in Proc. of 15th Int. Workshop on Distributed Systems: Operations and Management (DSOM'04), pp. 171--182, Nov. 2004."},{"key":"e_1_2_1_9_1","volume-title":"UAI'08","author":"Simma A.","year":"2008","unstructured":"A. Simma , M. Goldszmidt , J. MacCormick , P. Barham , R. Black , R. Isaacs , and R. Mortier , \" CT-NOR: representing and reasoning about events in continuous time \", In UAI'08 , July 2008 . A. Simma, M. Goldszmidt, J. MacCormick, P. Barham, R. Black, R. Isaacs, and R. Mortier, \"CT-NOR: representing and reasoning about events in continuous time\", In UAI'08, July 2008."},{"key":"e_1_2_1_10_1","unstructured":"Hadoop. http:\/\/hadoop.apache.org\/core.  Hadoop. http:\/\/hadoop.apache.org\/core."},{"key":"e_1_2_1_11_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementation","author":"Dean J.","year":"2004","unstructured":"J. Dean and S. Ghemawat , \" MapReduce: Simplified Data Processing on Large Clusters \", In USENIX Symposium on Operating Systems Design and Implementation , Dec. 2004 . J. Dean and S. Ghemawat, \"MapReduce: Simplified Data Processing on Large Clusters\", In USENIX Symposium on Operating Systems Design and Implementation, Dec. 2004."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/945445.945450"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2009.60"},{"key":"e_1_2_1_14_1","unstructured":"Hadoop bug reporting portal. http:\/\/issues.apache.org\/jira\/browse\/HADOOP  Hadoop bug reporting portal. http:\/\/issues.apache.org\/jira\/browse\/HADOOP"}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1740390.1740411","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1740390.1740411","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T12:40:55Z","timestamp":1750250455000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1740390.1740411"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,3,12]]},"references-count":14,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,3,12]]}},"alternative-id":["10.1145\/1740390.1740411"],"URL":"https:\/\/doi.org\/10.1145\/1740390.1740411","relation":{},"ISSN":["0163-5980"],"issn-type":[{"value":"0163-5980","type":"print"}],"subject":[],"published":{"date-parts":[[2010,3,12]]},"assertion":[{"value":"2010-03-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}