{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T03:20:15Z","timestamp":1777778415198,"version":"3.51.4"},"reference-count":59,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T00:00:00Z","timestamp":1760745600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"U.S Government","award":["Contract No. FA8702-19-C-0001"],"award-info":[{"award-number":["Contract No. FA8702-19-C-0001"]}]},{"name":"Rights in Technical Data-Noncommercial Items Clause","award":["DFARS 252.227-7013"],"award-info":[{"award-number":["DFARS 252.227-7013"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Information Visualization"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:p>There are many challenging decision-making situations due to high degrees of uncertainty, turbulence, information burden, interdependencies and competing decision criteria in complex environments such as military command and control and emergency response. Unfortunately, few decision support systems help these cognitively challenged decision makers understand the relative desirability of one course of action versus another, also known as option awareness (OA). We hypothesized that providing novel data visualization displays to compare options across multiple criteria in parallel, combined with interaction mechanisms tailored to exploring options, would improve decision makers\u2019 abilities to make better choices more confidently. To test our hypothesis, we created a design inspired by principles of OA and the forementioned challenges and instantiated it in a patented proof-of-concept implementation called the Multi-criteria Option Comparison Application (MOCA). We experimented with MOCA in command and control and emergency response scenarios with subject matter experts and obtained statistically significant quantitative data showing the participants believed their use of a MOCA-like OA application will result in more accurate and confident choices, an enhanced ability to consider multiple perspectives and relevant decision factors, and eight other benefits. Additionally, qualitative data analyses surfaced themes of innovation, discovery, collaboration, ambiguity reduction, assumption validation, and interactivity.<\/jats:p>","DOI":"10.1177\/14738716251377067","type":"journal-article","created":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T16:10:15Z","timestamp":1760803815000},"page":"168-191","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Multi-criteria option comparison in complex environments"],"prefix":"10.1177","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3111-4281","authenticated-orcid":false,"given":"Mark S.","family":"Pfaff","sequence":"first","affiliation":[{"name":"The MITRE Corporation, Bedford, MA, USA"}]},{"given":"Gary L.","family":"Klein","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA, USA"}]},{"given":"Brittany A.","family":"Tracy","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, McLean, VA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5783-9494","authenticated-orcid":false,"given":"Jill L.","family":"Drury","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, Bedford, MA, USA"}]},{"given":"Michelle","family":"Tsun","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, Bedford, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7570-2360","authenticated-orcid":false,"given":"Karina A.","family":"Roundtree","sequence":"additional","affiliation":[{"name":"The MITRE Corporation, Bedford, MA, USA"}]}],"member":"179","published-online":{"date-parts":[[2025,10,18]]},"reference":[{"key":"e_1_3_4_2_2","first-page":"789","volume-title":"Proceedings of the IEEE national aerospace and electronics conference","author":"Endsley MR","unstructured":"Endsley MR. Situation awareness global assessment technique (SAGAT). In: Proceedings of the IEEE national aerospace and electronics conference. IEEE, Dayton, OH, 23\u201327 May 1988, pp.789\u2013795."},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CogSIMA54611.2022.9830660"},{"key":"e_1_3_4_4_2","first-page":"537","volume-title":"IEEE conference on technologies for homeland security","author":"Drury JL","unstructured":"Drury JL, Klein G, Pfaff MS, et al. Dynamic decision support for emergency responders. In: IEEE conference on technologies for homeland security. Waltham, Massachusetts, 11\u201312 May 2009, pp.537\u2013544. IEEE."},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343412455799"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.protcy.2014.10.168"},{"key":"e_1_3_4_7_2","volume-title":"Proceedings of the 8th International Information Systems for Crisis Response and Management (ISCRAM) Conference","author":"Liu Y","unstructured":"Liu Y, Moon SP, Pfaff MS, et al. Collaborative option awareness for emergency response decision making. In: Proceedings of the 8th International Information Systems for Crisis Response and Management (ISCRAM) Conference, 2011 May. Belgium: ISCRAM."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.3.1.Johnson"},{"key":"e_1_3_4_9_2","volume-title":"Organizations in action: social science bases of administrative strategy","author":"Thompson JD","year":"1967","unstructured":"Thompson JD. Organizations in action: social science bases of administrative strategy. 1st ed. McGraw-Hill, 1967.","edition":"1"},{"key":"e_1_3_4_10_2","unstructured":"Klein GL Tracy BA Haines E et al. Multi-criteria option comparison tool. US Patent #12 153 785 B2. November 26 2024."},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39330-3_71"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2008.109"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.175"},{"key":"e_1_3_4_14_2","first-page":"249","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems","author":"Nielsen J","unstructured":"Nielsen J, Molich R. Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, Seattle, WA, 1990 March 1, pp.249\u2013256."},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2133416.2146416"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/38.974511"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1080\/01449298208914450"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/345513.345271"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05252-2_2"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1050.0451"},{"key":"e_1_3_4_21_2","volume-title":"Proceedings of the Army Science Conference","author":"Chandrasekaran B","unstructured":"Chandrasekaran B. Designing decision support systems to help avoid biases & make robust decisions, with examples from army planning. In: Proceedings of the Army Science Conference, DTIC, Orlando, FL, 1\u20134 December 2008."},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCDM.2007.369435"},{"key":"e_1_3_4_23_2","doi-asserted-by":"crossref","unstructured":"Kalra N Hallegatte S Lempert R et al. Agreeing on robust decisions: new processes for decision making under deep uncertainty. World Bank Policy Research Working Paper. 2014 (6906).","DOI":"10.1596\/1813-9450-6906"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhydrol.2013.03.006"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1021\/es506144f"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2015.07.014"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008325423033"},{"key":"e_1_3_4_28_2","volume-title":"Proceedings of the 10th International conference on information systems for crisis response and management (ISCRAM)","author":"Comfort LK","unstructured":"Comfort LK, Colella B, Voortman M, et al. Real-time decision making in urgent events: Modeling options for action. In: Proceedings of the 10th International conference on information systems for crisis response and management (ISCRAM), ISCRAM, Baden-Baden, Germany, 2013 May."},{"key":"e_1_3_4_29_2","volume-title":"Proceedings of the 7th International Information Systems for Crisis Response and Management (ISCRAM) Conference","author":"Mathieu J","unstructured":"Mathieu J, Pfaff MS, Drury JL, et al. Tactical robust decision-making methodology: effect of disease spread model fidelity on option awareness. In: Proceedings of the 7th International Information Systems for Crisis Response and Management (ISCRAM) Conference (eds French S, Tomaszewski BM, Zobel CW), Seattle, Washington, 2\u20135 May 2010. Belgium: ISCRAM."},{"key":"e_1_3_4_30_2","first-page":"1","volume-title":"Proceedings of the Naturalistic Decision Making (NDM) Conference","author":"Pfaff MS","unstructured":"Pfaff MS, Drury JL, Klein GL. Crowdsourcing mental models using DESIM (Descriptive to Executable Simulation Modeling). In: Proceedings of the Naturalistic Decision Making (NDM) Conference, McLean, Virginia, 9\u201312 June 2015, pp.1\u20137. Virginia: NDM."},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1201\/9780429459733-6"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199805000-00024"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1177\/0272989X08329463"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2013.01.034"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2015.01473"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1080\/10810730.2010.492560"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocz217"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343419842776"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1080\/1463922X.2017.1315750"},{"key":"e_1_3_4_40_2","doi-asserted-by":"crossref","unstructured":"Chen JY Procci K Boyce M et al. Situation awareness-based agent transparency. ARL-TR-6905. US Army Research Laboratory. 2014.","DOI":"10.21236\/ADA600351"},{"key":"e_1_3_4_41_2","first-page":"1","volume-title":"12th international command and control research and technology symposium","author":"Bruni S","year":"2007","unstructured":"Bruni S, Marquez JJ, Brzezinski A, et al. Introducing a human-automation collaboration taxonomy (HACT) in command and control decision-support systems. In: 12th international command and control research and technology symposium. Newport, MA: Command & Control Research Program, 2007, pp.1\u201313."},{"key":"e_1_3_4_42_2","first-page":"1","volume-title":"Proceedings of SPIE Defense and Security","author":"Ososky S","unstructured":"Ososky S, Sanders T, Jentsch F, et al. Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems. In: Proceedings of SPIE Defense and Security (eds Karlsen RE, Gage DW, Shoemaker CM, Gerhart GR), Baltimore, MD, USA, 5\u20139 May 2014, pp.1\u201312. Bellingham: SPIE."},{"key":"e_1_3_4_43_2","first-page":"127","volume-title":"Advances in human factors in robots and unmanned systems","author":"Lyons JB","year":"2016","unstructured":"Lyons JB, Sadler GG, Koltai K, et al. Shaping trust through transparent design: theoretical and experimental guidelines. In: Savage-Knapshield P, Chen JYC (eds) Advances in human factors in robots and unmanned systems. 1st ed. Springer, 2016, pp.127\u2013136.","edition":"1"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.41.3.435"},{"key":"e_1_3_4_45_2","volume-title":"Eurographics 2013\u2014state of the art reports","author":"Heinrich J","year":"2013","unstructured":"Heinrich J, Weiskopf D. State of the art of parallel coordinates. In: Sbert M, Szirmay-Kalos L (eds) Eurographics 2013\u2014state of the art reports. The Eurographics Association, 2013."},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01898350"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV.2009.25"},{"key":"e_1_3_4_48_2","volume-title":"Interactive edge-bundled parallel coordinates","author":"Li Z.","year":"2021","unstructured":"Li Z. Interactive edge-bundled parallel coordinates. Master\u2019s Thesis, University of Victoria, Canada, 2021."},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03129.x"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.visinf.2017.11.001"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/PacificVis.2014.40"},{"key":"e_1_3_4_52_2","unstructured":"Klein GA. Sources of power: how people make decisions. MIT Press 1998."},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470752937.ch15"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133216"},{"key":"e_1_3_4_55_2","volume-title":"Discovery of grounded theory: Strategies for qualitative research","author":"Glaser B","year":"1967","unstructured":"Glaser B, Strauss A. Discovery of grounded theory: Strategies for qualitative research. Aldine Publishing Co., 1967."},{"issue":"1","key":"e_1_3_4_56_2","first-page":"24","article-title":"Crisis management and multimedia technology in London underground line control rooms","volume":"1","author":"Heath C.","year":"1992","unstructured":"Heath C. Crisis management and multimedia technology in London underground line control rooms. J Comput Supp Coop Work 1992; 1(1): 24\u201348.","journal-title":"J Comput Supp Coop Work"},{"key":"e_1_3_4_57_2","volume-title":"Protocol analysis: verbal reports as data","author":"Simon HA","year":"1984","unstructured":"Simon HA, Ericsson KA. Protocol analysis: verbal reports as data. MIT Press, 1984."},{"key":"e_1_3_4_58_2","first-page":"76","volume-title":"Proceedings of international workshop on big data visual exploration and analytics (BigVis)","author":"Richer G","unstructured":"Richer G, Sansen J, Lalanne F, et al. Enabling hierarchical exploration for large-scale multidimensional data with abstract parallel coordinates. In: Proceedings of international workshop on big data visual exploration and analytics (BigVis), Vienna, Austria, March 2018, pp.76\u201383. BigVis."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598830"},{"key":"e_1_3_4_60_2","doi-asserted-by":"crossref","unstructured":"Aissi H Roy B. Robustness in multi-criteria decision aiding. In: Ehrgott M Figueira JR Greco S (eds) Trends in multiple criteria decision analysis. Springer 2010 pp.87\u2013121.","DOI":"10.1007\/978-1-4419-5904-1_4"}],"container-title":["Information Visualization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716251377067","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14738716251377067","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716251377067","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T19:19:27Z","timestamp":1777490367000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14738716251377067"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,18]]},"references-count":59,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1177\/14738716251377067"],"URL":"https:\/\/doi.org\/10.1177\/14738716251377067","relation":{},"ISSN":["1473-8716","1473-8724"],"issn-type":[{"value":"1473-8716","type":"print"},{"value":"1473-8724","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,18]]}}}