{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T22:28:08Z","timestamp":1762900088052,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":36,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T00:00:00Z","timestamp":1618358400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,14]]},"DOI":"10.1145\/3397481.3450658","type":"proceedings-article","created":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T06:07:18Z","timestamp":1618380438000},"page":"170-174","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML"],"prefix":"10.1145","author":[{"given":"Shweta","family":"Narkar","sequence":"first","affiliation":[{"name":"Tetherless World Constellation Rensselaer Polytechnic Institute, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunfeng","family":"Zhang","sequence":"additional","affiliation":[{"name":"IBM T. J. Watson Research Center, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Q. Vera","family":"Liao","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dakuo","family":"Wang","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin D.","family":"Weisz","sequence":"additional","affiliation":[{"name":"IBM Research AI, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,4,14]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346660"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Aaron Bangor Philip\u00a0T Kortum and James\u00a0T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human\u2013Computer Interaction 24 6(2008) 574\u2013594.  Aaron Bangor Philip\u00a0T Kortum and James\u00a0T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human\u2013Computer Interaction 24 6(2008) 574\u2013594.","DOI":"10.1080\/10447310802205776"},{"key":"e_1_3_2_1_4_1","volume-title":"FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE","author":"Brooks Michael","year":"2015","unstructured":"Michael Brooks , Saleema Amershi , Bongshin Lee , Steven\u00a0 M. Drucker , Ashish Kapoor , and Patrice Simard . 2015 . FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE , Chicago, IL, USA, 105\u2013112. https:\/\/doi.org\/10\/ggscqd Michael Brooks, Saleema Amershi, Bongshin Lee, Steven\u00a0M. Drucker, Ashish Kapoor, and Patrice Simard. 2015. FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, Chicago, IL, USA, 105\u2013112. https:\/\/doi.org\/10\/ggscqd"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2010.5652443"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Jaimie Drozdal Justin Weisz 2020. Exploring Information Needs for Establishing Trust in Automated Data Science Systems. In IUI\u201920. ACM in press.  Jaimie Drozdal Justin Weisz 2020. Exploring Information Needs for Establishing Trust in Automated Data Science Systems. In IUI\u201920. ACM in press.","DOI":"10.1145\/3377325.3377501"},{"key":"e_1_3_2_1_7_1","unstructured":"Google. [n.d.]. Cloud AutoML. Retrieved 3-April-2019 from https:\/\/cloud.google.com\/automl\/  Google. [n.d.]. Cloud AutoML. Retrieved 3-April-2019 from https:\/\/cloud.google.com\/automl\/"},{"key":"e_1_3_2_1_8_1","unstructured":"Google PAIR. 2018. What-If Tool. https:\/\/pair-code.github.io\/what-if-tool\/.  Google PAIR. 2018. What-If Tool. https:\/\/pair-code.github.io\/what-if-tool\/."},{"key":"e_1_3_2_1_9_1","unstructured":"H2O. [n.d.]. H2O. Retrieved 3-April-2019 from https:\/\/h2o.ai  H2O. [n.d.]. H2O. Retrieved 3-April-2019 from https:\/\/h2o.ai"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300809"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134688"},{"key":"e_1_3_2_1_12_1","unstructured":"IBM. [n.d.]. AutoAI. Retrieved 06-Oct-2019 from https:\/\/www.ibm.com\/cloud\/watson-studio\/autoai  IBM. [n.d.]. AutoAI. Retrieved 06-Oct-2019 from https:\/\/www.ibm.com\/cloud\/watson-studio\/autoai"},{"key":"e_1_3_2_1_13_1","volume-title":"Activis: Visual exploration of industry-scale deep neural network models","author":"Kahng Minsuk","year":"2017","unstructured":"Minsuk Kahng , Pierre\u00a0 Y Andrews , Aditya Kalro , and Duen Horng\u00a0Polo Chau . 2017 . Activis: Visual exploration of industry-scale deep neural network models . IEEE transactions on visualization and computer graphics 24, 1(2017), 88\u201397. Minsuk Kahng, Pierre\u00a0Y Andrews, Aditya Kalro, and Duen Horng\u00a0Polo Chau. 2017. Activis: Visual exploration of industry-scale deep neural network models. IEEE transactions on visualization and computer graphics 24, 1(2017), 88\u201397."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_3_2_1_15_1","volume-title":"Magic Quadrant for data science and machine-learning platforms. Gartner","author":"Krensky Peter","year":"2020","unstructured":"Peter Krensky , Pieter\u00a0den Harner , Erick Brethenoux , Jim Hare , Svetlana Sicular , and Shubhangi Vashisth . 2020. Magic Quadrant for data science and machine-learning platforms. Gartner , Inc ( 2020 ). Peter Krensky, Pieter\u00a0den Harner, Erick Brethenoux, Jim Hare, Svetlana Sicular, and Shubhangi Vashisth. 2020. Magic Quadrant for data science and machine-learning platforms. Gartner, Inc (2020)."},{"key":"e_1_3_2_1_16_1","volume-title":"A Human-in-the-loop Perspective on AutoML: Milestones and the Road Ahead. Data Engineering","author":"Jung-Lin Lee Doris","year":"2019","unstructured":"Doris Jung-Lin Lee , Stephen Macke , Doris Xin , Angela Lee , Silu Huang , and Aditya Parameswaran . 2019. A Human-in-the-loop Perspective on AutoML: Milestones and the Road Ahead. Data Engineering ( 2019 ), 58. Doris Jung-Lin Lee, Stephen Macke, Doris Xin, Angela Lee, Silu Huang, and Aditya Parameswaran. 2019. A Human-in-the-loop Perspective on AutoML: Milestones and the Road Ahead. Data Engineering (2019), 58."},{"key":"e_1_3_2_1_17_1","unstructured":"LendingClub. [n.d.]. LendingClub Statistics. https:\/\/www.lendingclub.com\/info\/statistics.action  LendingClub. [n.d.]. LendingClub Statistics. https:\/\/www.lendingclub.com\/info\/statistics.action"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Sijia Liu Parikshit Ram Deepak Vijaykeerthy Djallel Bouneffouf Gregory Bramble Horst Samulowitz Dakuo Wang Andrew Conn and Alexander\u00a0G Gray. 2020. An ADMM Based Framework for AutoML Pipeline Configuration.. In AAAI. 4892\u20134899.  Sijia Liu Parikshit Ram Deepak Vijaykeerthy Djallel Bouneffouf Gregory Bramble Horst Samulowitz Dakuo Wang Andrew Conn and Alexander\u00a0G Gray. 2020. An ADMM Based Framework for AutoML Pipeline Configuration.. In AAAI. 4892\u20134899.","DOI":"10.1609\/aaai.v34i04.5926"},{"key":"e_1_3_2_1_19_1","first-page":"I","article-title":"A Unified Approach to Interpreting Model Predictions","volume":"30","author":"Lundberg M","year":"2017","unstructured":"Scott\u00a0 M Lundberg and Su-In Lee . 2017 . A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 , I .\u00a0Guyon, U.\u00a0V. Luxburg, S.\u00a0Bengio, H.\u00a0Wallach, R.\u00a0Fergus, S.\u00a0Vishwanathan, and R.\u00a0Garnett(Eds.). Curran Associates, Inc., 4765\u20134774. Scott\u00a0M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I.\u00a0Guyon, U.\u00a0V. Luxburg, S.\u00a0Bengio, H.\u00a0Wallach, R.\u00a0Fergus, S.\u00a0Vishwanathan, and R.\u00a0Garnett(Eds.). Curran Associates, Inc., 4765\u20134774.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the 2020 ACM conference on GROUP. ACM.","author":"Mao Yaoli","year":"2020","unstructured":"Yaoli Mao , Dakuo Wang , Michael Muller , Kush Varshney , Ioana Baldini , Casey Dugan , and Aleksandra Mojsilovic . 2020 . How Data Scientists Work Together With Domain Experts in Scientific Collaborations . In Proceedings of the 2020 ACM conference on GROUP. ACM. Yaoli Mao, Dakuo Wang, Michael Muller, Kush Varshney, Ioana Baldini, Casey Dugan, and Aleksandra Mojsilovic. 2020. How Data Scientists Work Together With Domain Experts in Scientific Collaborations. In Proceedings of the 2020 ACM conference on GROUP. ACM."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449205"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Donghao Ren Saleema Amershi Bongshin Lee Jina Suh and Jason\u00a0D. Williams. [n.d.]. Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers. 23 1([n.\u00a0d.]) 61\u201370. https:\/\/doi.org\/10\/f9zx4t  Donghao Ren Saleema Amershi Bongshin Lee Jina Suh and Jason\u00a0D. Williams. [n.d.]. Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers. 23 1([n.\u00a0d.]) 61\u201370. https:\/\/doi.org\/10\/f9zx4t","DOI":"10.1109\/TVCG.2016.2598828"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744158"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-013-0679-x"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376866"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445526"},{"key":"e_1_3_2_1_30_1","unstructured":"Dakuo Wang Q.\u00a0Vera Liao Yunfeng Zhang Udayan Khurana Horst Samulowitz Soya Park Michael Muller and Lisa Amini. 2021. How Much Automation Does a Data Scientist Want?. In pre-print.  Dakuo Wang Q.\u00a0Vera Liao Yunfeng Zhang Udayan Khurana Horst Samulowitz Soya Park Michael Muller and Lisa Amini. 2021. How Much Automation Does a Data Scientist Want?. In pre-print."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379336.3381474"},{"key":"e_1_3_2_1_32_1","volume-title":"Perceptions of Automated AI. To appear in Computer Supported Cooperative Work (CSCW)","author":"Wang Dakuo","year":"2019","unstructured":"Dakuo Wang , Justin\u00a0 D. Weisz , Michael Muller , Parikshit Ram , Werner Geyer , Casey Dugan , Yla Tausczik , Horst Samulowitz , and Alexander Gray . 2019. Human-AI Collaboration in Data Science: Exploring Data Scientists \u2019 Perceptions of Automated AI. To appear in Computer Supported Cooperative Work (CSCW) ( 2019 ). Dakuo Wang, Justin\u00a0D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI Collaboration in Data Science: Exploring Data Scientists\u2019 Perceptions of Automated AI. To appear in Computer Supported Cooperative Work (CSCW) (2019)."},{"volume-title":"ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk","author":"Wang Qianwen","key":"e_1_3_2_1_33_1","unstructured":"Qianwen Wang , Yao Ming , Zhihua Jin , Qiaomu Shen , Dongyu Liu , Micah\u00a0 J. Smith , Kalyan Veeramachaneni , and Huamin Qu. [n.d.]. ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk , 2019-05-02) (CHI \u201919). Association for Computing Machinery, 1\u201312. https:\/\/doi.org\/10\/ggcn2s Qianwen Wang, Yao Ming, Zhihua Jin, Qiaomu Shen, Dongyu Liu, Micah\u00a0J. Smith, Kalyan Veeramachaneni, and Huamin Qu. [n.d.]. ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk, 2019-05-02) (CHI \u201919). Association for Computing Machinery, 1\u201312. https:\/\/doi.org\/10\/ggcn2s"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Daniel Weidele Justin Weisz Erick Oduor Michael Muller Josh Andres Alexander Gray and Dakuo Wang. 2020. AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates. In IUI\u201920. ACM in press.  Daniel Weidele Justin Weisz Erick Oduor Michael Muller Josh Andres Alexander Gray and Dakuo Wang. 2020. AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates. In IUI\u201920. ACM in press.","DOI":"10.1145\/3377325.3377538"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Jiawei Zhang Yang Wang Piero Molino Lezhi Li and David\u00a0S. Ebert. [n.d.]. Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models. 25 1([n.\u00a0d.]) 364\u2013373. https:\/\/doi.org\/10\/ggsr89 arxiv:1808.00196  Jiawei Zhang Yang Wang Piero Molino Lezhi Li and David\u00a0S. Ebert. [n.d.]. Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models. 25 1([n.\u00a0d.]) 364\u2013373. https:\/\/doi.org\/10\/ggsr89 arxiv:1808.00196","DOI":"10.1109\/TVCG.2018.2864499"},{"key":"e_1_3_2_1_36_1","unstructured":"Marc-Andr\u00e9 Z\u00f6ller and Marco\u00a0F Huber. 2019. Survey on Automated Machine Learning. arXiv preprint arXiv:1904.12054(2019).  Marc-Andr\u00e9 Z\u00f6ller and Marco\u00a0F Huber. 2019. Survey on Automated Machine Learning. arXiv preprint arXiv:1904.12054(2019)."}],"event":{"name":"IUI '21: 26th International Conference on Intelligent User Interfaces","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"College Station TX USA","acronym":"IUI '21"},"container-title":["26th International Conference on Intelligent User Interfaces"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397481.3450658","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397481.3450658","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:33Z","timestamp":1750200093000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397481.3450658"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,14]]},"references-count":36,"alternative-id":["10.1145\/3397481.3450658","10.1145\/3397481"],"URL":"https:\/\/doi.org\/10.1145\/3397481.3450658","relation":{},"subject":[],"published":{"date-parts":[[2021,4,14]]},"assertion":[{"value":"2021-04-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}