{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T16:49:22Z","timestamp":1771606162355,"version":"3.50.1"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2016,3,17]],"date-time":"2016-03-17T00:00:00Z","timestamp":1458172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Adobe and the University of Maryland\/Mpowering the State through the Center for Health-related Informatics and Bioimaging"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2016,5,5]]},"abstract":"<jats:p>Cohort comparison studies have traditionally been hypothesis driven and conducted in carefully controlled environments (such as clinical trials). Given two groups of event sequence data, researchers test a single hypothesis (e.g., does the group taking Medication A exhibit more deaths than the group taking Medication B?). Recently, however, researchers have been moving toward more exploratory methods of retrospective analysis with existing data. In this article, we begin by showing that the task of cohort comparison is specific enough to support automatic computation against a bounded set of potential questions and objectives, a method that we refer to as High-Volume Hypothesis Testing (HVHT). From this starting point, we demonstrate that the diversity of these objectives, both across and within different domains, as well as the inherent complexities of real-world datasets, still requires human involvement to determine meaningful insights. We explore how visualization and interaction better support the task of exploratory data analysis and the understanding of HVHT results (how significant they are, why they are meaningful, and whether the entire dataset has been exhaustively explored). Through interviews and case studies with domain experts, we iteratively design and implement visualization and interaction techniques in a visual analytics tool, CoCo. As a result of our evaluation, we propose six design guidelines for enabling users to explore large result sets of HVHT systematically and flexibly in order to glean meaningful insights more quickly. Finally, we illustrate the utility of this method with three case studies in the medical domain.<\/jats:p>","DOI":"10.1145\/2890478","type":"journal-article","created":{"date-parts":[[2016,3,18]],"date-time":"2016-03-18T13:50:44Z","timestamp":1458309044000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":33,"title":["High-Volume Hypothesis Testing"],"prefix":"10.1145","volume":"6","author":[{"given":"Sana","family":"Malik","sequence":"first","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ben","family":"Shneiderman","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Du","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Catherine","family":"Plaisant","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Margret","family":"Bjarnadottir","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,3,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/170035.170072"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/pds.1230"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011429418057"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013699998"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1186\/cc2955"},{"key":"e_1_2_1_6_1","first-page":"2","article-title":"Understanding adherence and prescription patterns using large scale claims data","volume":"34","author":"Bjarnadottir Margret","year":"2015","unstructured":"Margret Bjarnadottir , Sana Malik , Eberechukwu Onukwugha , Tanisha Gooden , and Catherine Plaisant . 2015 . Understanding adherence and prescription patterns using large scale claims data . Pharmacoeconomics 34 , 2 (Feb. 2016), 169--79. DOI:10.1007\/s40273-015-0333-4 10.1007\/s40273-015-0333-4 Margret Bjarnadottir, Sana Malik, Eberechukwu Onukwugha, Tanisha Gooden, and Catherine Plaisant. 2015. Understanding adherence and prescription patterns using large scale claims data. Pharmacoeconomics 34, 2 (Feb. 2016), 169--79. DOI:10.1007\/s40273-015-0333-4","journal-title":"Pharmacoeconomics"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Workshop on Interactive Systems in Healthcare","author":"Carter Elizabeth","year":"2013","unstructured":"Elizabeth Carter , Randall Burd , Megan Monroe , Catherine Plaisant , and Ben Shneiderman . 2013 . Using eventflow to analyze task performance during trauma resuscitation . In Proceedings of the Workshop on Interactive Systems in Healthcare (2013). Elizabeth Carter, Randall Burd, Megan Monroe, Catherine Plaisant, and Ben Shneiderman. 2013. Using eventflow to analyze task performance during trauma resuscitation. In Proceedings of the Workshop on Interactive Systems in Healthcare (2013)."},{"key":"e_1_2_1_8_1","volume-title":"Modelling Survival Data in Medical Research","author":"Collett David","unstructured":"David Collett . 2003. Modelling Survival Data in Medical Research , 2 nd ed. Chapman and Hall\/CRC Press . David Collett. 2003. Modelling Survival Data in Medical Research, 2nd ed. Chapman and Hall\/CRC Press.","edition":"2"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1378\/chest.125.5.1815"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201915)","author":"Federico Paolo","year":"2015","unstructured":"Paolo Federico , J\u00fcrgen Unger , Albert Amor-Amors , Lucia Sacchi , Denis Klimov , and Silvia Miksch . 2015 . Gnaeus: Utilizing clinical guidelines for knowledge-assisted visualisation of EHR cohorts . In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201915) , Enrico Bertini and Jonathan C. Roberts (Eds.). The Eurographics Association. DOI:http:\/\/dx.doi.org\/10.2312\/eurova. 20151108 10.2312\/eurova.20151108 Paolo Federico, J\u00fcrgen Unger, Albert Amor-Amors, Lucia Sacchi, Denis Klimov, and Silvia Miksch. 2015. Gnaeus: Utilizing clinical guidelines for knowledge-assisted visualisation of EHR cohorts. In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201915), Enrico Bertini and Jonathan C. Roberts (Eds.). The Eurographics Association. DOI:http:\/\/dx.doi.org\/10.2312\/eurova.20151108"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/1473871611416549"},{"key":"e_1_2_1_13_1","first-page":"4","article-title":"Understanding survival analysis: Kaplan-Meier estimate","volume":"1","author":"Goel Manish K.","year":"2010","unstructured":"Manish K. Goel , Pardeep Khanna , and Jugal Kishore . 2010 . Understanding survival analysis: Kaplan-Meier estimate . International Journal of Ayurveda Research 1 , 4 (October 2010), 274--278. DOI:http:\/\/dx.doi.org\/10.4103\/0974-7788.76794 10.4103\/0974-7788.76794 Manish K. Goel, Pardeep Khanna, and Jugal Kishore. 2010. Understanding survival analysis: Kaplan-Meier estimate. International Journal of Ayurveda Research 1, 4 (October 2010), 274--278. DOI:http:\/\/dx.doi.org\/10.4103\/0974-7788.76794","journal-title":"International Journal of Ayurveda Research"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Transportation Research Board 90th Annual Meeting.","author":"Guerra-G\u00f3mez John Alexis","year":"2011","unstructured":"John Alexis Guerra-G\u00f3mez , Krist Wongsuphasawat , Taowei David Wang , Michael L. Pack , and Catherine Plaisant . 2011 . Analyzing incident management event sequences with interactive visualization . In Proceedings of the Transportation Research Board 90th Annual Meeting. John Alexis Guerra-G\u00f3mez, Krist Wongsuphasawat, Taowei David Wang, Michael L. Pack, and Catherine Plaisant. 2011. Analyzing incident management event sequences with interactive visualization. In Proceedings of the Transportation Research Board 90th Annual Meeting."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-006-0059-1"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/dji139"},{"key":"e_1_2_1_17_1","first-page":"5","article-title":"The cost of treating skeletal-related events in patients with prostate cancer","volume":"14","author":"Lage Maureen J.","year":"2008","unstructured":"Maureen J. Lage , Beth L. Barber , David J. Harrison , and Sun Jun . 2008 . The cost of treating skeletal-related events in patients with prostate cancer . American Journal of Managed Care 14 , 5 (May 2008), 317--322. Maureen J. Lage, Beth L. Barber, David J. Harrison, and Sun Jun. 2008. The cost of treating skeletal-related events in patients with prostate cancer. American Journal of Managed Care 14, 5 (May 2008), 317--322.","journal-title":"American Journal of Managed Care"},{"key":"e_1_2_1_18_1","volume-title":"Special section on visual analytics: Mind the time: Unleashing temporal aspects in pattern discovery. Computer Graphics 38 (February","author":"Lammarsch Tim","year":"2014","unstructured":"Tim Lammarsch , Wolfgang Aigner , Alessio Bertone , Silvia Miksch , and Alexander Rind . 2014. Special section on visual analytics: Mind the time: Unleashing temporal aspects in pattern discovery. Computer Graphics 38 (February 2014 ), 38--50. DOI:http:\/\/dx.doi.org\/10.1016\/j.cag.2013.10.007 10.1016\/j.cag.2013.10.007 Tim Lammarsch, Wolfgang Aigner, Alessio Bertone, Silvia Miksch, and Alexander Rind. 2014. Special section on visual analytics: Mind the time: Unleashing temporal aspects in pattern discovery. Computer Graphics 38 (February 2014), 38--50. DOI:http:\/\/dx.doi.org\/10.1016\/j.cag.2013.10.007"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767907"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2014.55.1143"},{"key":"e_1_2_1_21_1","volume-title":"Ezeife","author":"Mabroukeh Nizar R.","year":"2010","unstructured":"Nizar R. Mabroukeh and Christie I . Ezeife . 2010 . A taxonomy of sequential pattern mining algorithms. Computer Surveys 43, 1, Article 3 (Nov. 2010), 3:1--3:41 pages. DOI:http:\/\/dx.doi.org\/10.1145\/1824795.1824798 10.1145\/1824795.1824798 Nizar R. Mabroukeh and Christie I. Ezeife. 2010. A taxonomy of sequential pattern mining algorithms. Computer Surveys 43, 1, Article 3 (Nov. 2010), 3:1--3:41 pages. DOI:http:\/\/dx.doi.org\/10.1145\/1824795.1824798"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701407"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.200"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.21236\/ADA588017"},{"key":"e_1_2_1_26_1","first-page":"1","article-title":"Skeletal related events, bone metastasis and survival of prostate cancer: A population based cohort study in denmark (1999 to 2007)","volume":"184","author":"N\u00f8rgaard Mette","year":"2014","unstructured":"Mette N\u00f8rgaard , Annette \u00d8stergaard Jensen , Jacob Bonde Jacobsen , Kara Cetin , Jon P. Fryzek , and Henrik Toft S\u00f8rensen . 2014 . Skeletal related events, bone metastasis and survival of prostate cancer: A population based cohort study in denmark (1999 to 2007) . Journal of Urology 184 , 1 (October 6, 2015), 162--167. DOI:http:\/\/dx.doi.org\/10.1016\/j.juro.2010.03.034 10.1016\/j.juro.2010.03.034 Mette N\u00f8rgaard, Annette \u00d8stergaard Jensen, Jacob Bonde Jacobsen, Kara Cetin, Jon P. Fryzek, and Henrik Toft S\u00f8rensen. 2014. Skeletal related events, bone metastasis and survival of prostate cancer: A population based cohort study in denmark (1999 to 2007). Journal of Urology 184, 1 (October 6, 2015), 162--167. DOI:http:\/\/dx.doi.org\/10.1016\/j.juro.2010.03.034","journal-title":"Journal of Urology"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2557500.2557508"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.4103\/0970-2113.106127"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.213"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ps.46.020195.003021"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1168149.1168158"},{"key":"e_1_2_1_33_1","unstructured":"Tableau Software. 2014. Tableau. Retrieved from http:\/\/www.tableausoftware.com\/.  Tableau Software. 2014. Tableau. Retrieved from http:\/\/www.tableausoftware.com\/."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346574"},{"key":"e_1_2_1_35_1","unstructured":"TIBCO. 2014. Spotfire. Retrieved from http:\/\/spotfire.tibco.com\/.  TIBCO. 2014. Spotfire. Retrieved from http:\/\/spotfire.tibco.com\/."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2014.7042532"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/1473871613477852"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1882992.1883001"},{"key":"e_1_2_1_39_1","first-page":"4","article-title":"Iterative cohort analysis and exploration","volume":"14","author":"Zhang Zhiyuan","year":"2014","unstructured":"Zhiyuan Zhang , David Gotz , and Adam Perer . 2014 . Iterative cohort analysis and exploration . Information Visualization 14 , 4 (Oct. 2015), 289--307. Zhiyuan Zhang, David Gotz, and Adam Perer. 2014. Iterative cohort analysis and exploration. Information Visualization 14, 4 (Oct. 2015), 289--307.","journal-title":"Information Visualization"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702419"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2890478","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2890478","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:38:55Z","timestamp":1750221535000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2890478"}},"subtitle":["Systematic Exploration of Event Sequence Comparisons"],"short-title":[],"issued":{"date-parts":[[2016,3,17]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,5,5]]}},"alternative-id":["10.1145\/2890478"],"URL":"https:\/\/doi.org\/10.1145\/2890478","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,3,17]]},"assertion":[{"value":"2015-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}