{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T14:36:46Z","timestamp":1776091006938,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":69,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T00:00:00Z","timestamp":1651190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,29]]},"DOI":"10.1145\/3491102.3502102","type":"proceedings-article","created":{"date-parts":[[2022,4,28]],"date-time":"2022-04-28T16:33:27Z","timestamp":1651163607000},"page":"1-14","source":"Crossref","is-referenced-by-count":39,"title":["Symphony: Composing Interactive Interfaces for Machine Learning"],"prefix":"10.1145","author":[{"given":"Alex","family":"B\u00e4uerle","sequence":"first","affiliation":[{"name":"Visual Computing Group, Ulm University, Germany and Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"\u00c1ngel Alexander","family":"Cabrera","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, United States and Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fred","family":"Hohman","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Megan","family":"Maher","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Koski","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xavier","family":"Suau","sequence":"additional","affiliation":[{"name":"Apple, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Titus","family":"Barik","sequence":"additional","affiliation":[{"name":"Apple, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dominik","family":"Moritz","sequence":"additional","affiliation":[{"name":"Apple, United States and Human-Computer Interaction Institute, Carnegie Mellon University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,4,29]]},"reference":[{"key":"e_1_3_2_2_1_1","first-page":"1086","article-title":"Fairsight: Visual analytics for fairness in decision making","volume":"26","author":"Ahn Yongsu","year":"2019","unstructured":"Yongsu Ahn and Yu-Ru Lin . 2019 . Fairsight: Visual analytics for fairness in decision making . IEEE Transactions on Visualization and Computer Graphics 26 , 1(2019), 1086 \u2013 1095 . Yongsu Ahn and Yu-Ru Lin. 2019. Fairsight: Visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics 26, 1(2019), 1086\u20131095.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_3_2_2_2_1","volume-title":"IEEE Symposium on Information Visualization, INFO VIS. IEEE, 111\u2013117","author":"Amar Robert","year":"2005","unstructured":"Robert Amar , James Eagan , and John Stasko . 2005 . Low-level components of analytic activity in information visualization . In IEEE Symposium on Information Visualization, INFO VIS. IEEE, 111\u2013117 . Robert Amar, James Eagan, and John Stasko. 2005. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, INFO VIS. IEEE, 111\u2013117."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2942288"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2743990"},{"key":"e_1_3_2_2_6_1","volume-title":"Computer Graphics Forum, Vol.\u00a039","author":"B\u00e4uerle Alex","unstructured":"Alex B\u00e4uerle , Heiko Neumann , and Timo Ropinski . 2020. Classifier-guided visual correction of noisy labels for image classification tasks . In Computer Graphics Forum, Vol.\u00a039 . Wiley Online Library , 195\u2013205. Alex B\u00e4uerle, Heiko Neumann, and Timo Ropinski. 2020. Classifier-guided visual correction of noisy labels for image classification tasks. In Computer Graphics Forum, Vol.\u00a039. Wiley Online Library, 195\u2013205."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00041"},{"key":"e_1_3_2_2_8_1","volume-title":"Conference on Fairness, Accountability and Transparency. PMLR, 77\u201391","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on Fairness, Accountability and Transparency. PMLR, 77\u201391 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency. PMLR, 77\u201391."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST47406.2019.8986948"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934261"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2020.2973258"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172950"},{"key":"e_1_3_2_2_13_1","unstructured":"European Commission. 2019. Ethics guidelines for trustworthy AI. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/ethics-guidelines-trustworthy-ai  European Commission. 2019. Ethics guidelines for trustworthy AI. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/ethics-guidelines-trustworthy-ai"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445267"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0055-y"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474734"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300500"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01219-9_47"},{"key":"e_1_3_2_2_20_1","volume-title":"Confusionflow: A model-agnostic visualization for temporal analysis of classifier confusion","author":"Hinterreiter Andreas","year":"2020","unstructured":"Andreas Hinterreiter , Peter Ruch , Holger Stitz , Martin Ennemoser , Jurgen Bernard , Hendrik Strobelt , and Marc Streit . 2020 . Confusionflow: A model-agnostic visualization for temporal analysis of classifier confusion . IEEE Transactions on Visualization and Computer Graphics ( 2020). Andreas Hinterreiter, Peter Ruch, Holger Stitz, Martin Ennemoser, Jurgen Bernard, Hendrik Strobelt, and Marc Streit. 2020. Confusionflow: A model-agnostic visualization for temporal analysis of classifier confusion. IEEE Transactions on Visualization and Computer Graphics (2020)."},{"key":"e_1_3_2_2_21_1","volume-title":"Visual analytics in deep learning: An interrogative survey for the next frontiers","author":"Hohman Fred","year":"2018","unstructured":"Fred Hohman , Minsuk Kahng , Robert Pienta , and Duen\u00a0Horng Chau . 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers . IEEE Transactions on Visualization and Computer Graphics ( 2018 ). https:\/\/doi.org\/10.1109\/TVCG.2018.2843369 Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen\u00a0Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics (2018). https:\/\/doi.org\/10.1109\/TVCG.2018.2843369"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934659"},{"key":"e_1_3_2_2_23_1","unstructured":"Sarah Holland Ahmed Hosny Sarah Newman Joshua Joseph and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677(2018).  Sarah Holland Ahmed Hosny Sarah Newman Joshua Joseph and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677(2018)."},{"key":"e_1_3_2_2_24_1","unstructured":"Holoviz. 2021. Panel. https:\/\/panel.holoviz.org\/  Holoviz. 2021. Panel. https:\/\/panel.holoviz.org\/"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2007.55"},{"key":"e_1_3_2_2_26_1","unstructured":"DataBricks Inc.2021. DataBricks. https:\/\/databricks.com\/  DataBricks Inc.2021. DataBricks. https:\/\/databricks.com\/"},{"key":"e_1_3_2_2_27_1","unstructured":"Google Inc.2021. Facets. https:\/\/pair-code.github.io\/facets\/  Google Inc.2021. Facets. https:\/\/pair-code.github.io\/facets\/"},{"key":"e_1_3_2_2_28_1","unstructured":"Google Inc.2021. Know Your Data. https:\/\/knowyourdata.withgoogle.com\/  Google Inc.2021. Know Your Data. https:\/\/knowyourdata.withgoogle.com\/"},{"key":"e_1_3_2_2_29_1","unstructured":"Observable Inc.2021. Observable. https:\/\/observablehq.com\/  Observable Inc.2021. Observable. https:\/\/observablehq.com\/"},{"key":"e_1_3_2_2_30_1","unstructured":"Plotly\u00a0Technologies Inc.2015. Collaborative data science. Montreal QC. https:\/\/plot.ly  Plotly\u00a0Technologies Inc.2015. Collaborative data science. Montreal QC. https:\/\/plot.ly"},{"key":"e_1_3_2_2_31_1","unstructured":"Streamlit Inc.2021. Streamlit. https:\/\/streamlit.io\/  Streamlit Inc.2021. Streamlit. https:\/\/streamlit.io\/"},{"key":"e_1_3_2_2_32_1","unstructured":"Jupyter. 2021. IPyWidgets. https:\/\/ipywidgets.readthedocs.io\/en\/stable\/  Jupyter. 2021. IPyWidgets. https:\/\/ipywidgets.readthedocs.io\/en\/stable\/"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744718"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939503"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415842"},{"key":"e_1_3_2_2_37_1","volume-title":"Positioning and Power in Academic Publishing: Players, Agents and Agendas, F.\u00a0Loizides and B.\u00a0Schmidt (Eds.)","author":"Kluyver Thomas","unstructured":"Thomas Kluyver , Benjamin Ragan-Kelley , Fernando P\u00e9rez , Brian Granger , Matthias Bussonnier , Jonathan Frederic , Kyle Kelley , Jessica Hamrick , Jason Grout , Sylvain Corlay , Paul Ivanov , Dami\u00e1n Avila , Safia Abdalla , and Carol Willing . 2016. Jupyter Notebooks \u2013 a publishing format for reproducible computational workflows . In Positioning and Power in Academic Publishing: Players, Agents and Agendas, F.\u00a0Loizides and B.\u00a0Schmidt (Eds.) . IOS Press , 87 \u2013 90. Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P\u00e9rez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Dami\u00e1n Avila, Safia Abdalla, and Carol Willing. 2016. Jupyter Notebooks \u2013 a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas, F.\u00a0Loizides and B.\u00a0Schmidt (Eds.). IOS Press, 87 \u2013 90."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300330"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_3_2_2_40_1","unstructured":"Alex Krizhevsky Geoffrey Hinton 2009. Learning multiple layers of features from tiny images. (2009).  Alex Krizhevsky Geoffrey Hinton 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_2_2_41_1","first-page":"727","article-title":"Lux: Always-on visualization recommendations for exploratory data science","volume":"15","author":"Jung-Lin Lee Doris","year":"2022","unstructured":"Doris Jung-Lin Lee , Dixin Tang , Kunal Agarwal , Thyne Boonmark , Caitlyn Chen , Jake Kang , Ujjaini Mukhopadhyay , Jerry Song , Micah Yong , Marti\u00a0 A. Hearst , and Aditya\u00a0 G. Parameswaran . 2022 . Lux: Always-on visualization recommendations for exploratory data science . Proceedings of the VLDB Endowment 15 , 3 (2022), 727 \u2013 738 . Doris Jung-Lin Lee, Dixin Tang, Kunal Agarwal, Thyne Boonmark, Caitlyn Chen, Jake Kang, Ujjaini Mukhopadhyay, Jerry Song, Micah Yong, Marti\u00a0A. Hearst, and Aditya\u00a0G. Parameswaran. 2022. Lux: Always-on visualization recommendations for exploratory data science. Proceedings of the VLDB Endowment 15, 3 (2022), 727\u2013738.","journal-title":"Proceedings of the VLDB Endowment"},{"key":"e_1_3_2_2_42_1","unstructured":"Fritz Lekschas. 2021. Regl Scatterplot. https:\/\/github.com\/flekschas\/regl-scatterplot  Fritz Lekschas. 2021. Regl Scatterplot. https:\/\/github.com\/flekschas\/regl-scatterplot"},{"key":"e_1_3_2_2_43_1","unstructured":"Tableau\u00a0Software LLC. 2021. Tableau. https:\/\/www.tableau.com\/  Tableau\u00a0Software LLC. 2021. Tableau. https:\/\/www.tableau.com\/"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368555.3384468"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866038"},{"key":"e_1_3_2_2_49_1","unstructured":"Plotly. 2021. Dash. https:\/\/plotly.com\/dash\/  Plotly. 2021. Dash. https:\/\/plotly.com\/dash\/"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598828"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.01.105"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467091"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419764"},{"key":"e_1_3_2_2_54_1","volume-title":"Notes from the AI frontier: Tackling bias in AI (and in humans)","author":"Silberg Jake","year":"2019","unstructured":"Jake Silberg and James Manyika . 2019. Notes from the AI frontier: Tackling bias in AI (and in humans) . McKinsey Global Institute (June 2019 )(2019). Jake Silberg and James Manyika. 2019. Notes from the AI frontier: Tackling bias in AI (and in humans). McKinsey Global Institute (June 2019)(2019)."},{"key":"e_1_3_2_2_55_1","volume-title":"Amazon\u2019s face recognition falsely matched 28 members of congress with mugshots","author":"Snow Jacob","year":"2018","unstructured":"Jacob Snow . 2018. Amazon\u2019s face recognition falsely matched 28 members of congress with mugshots . American Civil Liberties Union 28 ( 2018 ). Jacob Snow. 2018. Amazon\u2019s face recognition falsely matched 28 members of congress with mugshots. American Civil Liberties Union 28 (2018)."},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1177\/030631289019003001"},{"key":"e_1_3_2_2_57_1","volume-title":"Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models","author":"Strobelt Hendrik","year":"2018","unstructured":"Hendrik Strobelt , Sebastian Gehrmann , Michael Behrisch , Adam Perer , Hanspeter Pfister , and Alexander\u00a0 M. Rush . 2019. Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models . In IEEE Transactions on Visualization and Computer Graphics, Vol .\u00a025. 353\u2013363. https:\/\/doi.org\/10.1109\/TVCG. 2018 .2865044 Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander\u00a0M. Rush. 2019. Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models. In IEEE Transactions on Visualization and Computer Graphics, Vol.\u00a025. 353\u2013363. https:\/\/doi.org\/10.1109\/TVCG.2018.2865044"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01057"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04051-w"},{"key":"e_1_3_2_2_60_1","unstructured":"Voila. 2021. Voila. https:\/\/github.com\/voila-dashboards\/voila  Voila. 2021. Voila. https:\/\/github.com\/voila-dashboards\/voila"},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58580-8_43"},{"key":"e_1_3_2_2_62_1","first-page":"56","article-title":"The what-if tool: Interactive probing of machine learning models","volume":"26","author":"Wexler James","year":"2020","unstructured":"James Wexler , Mahima Pushkarna , Tolga Bolukbasi , Martin Wattenberg , Fernanda Viegas , and Jimbo Wilson . 2020 . The what-if tool: Interactive probing of machine learning models . IEEE Transactions on Visualization and Computer Graphics 26 (2020), 56 \u2013 65 . Issue 1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934619 James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viegas, and Jimbo Wilson. 2020. The what-if tool: Interactive probing of machine learning models. IEEE Transactions on Visualization and Computer Graphics 26 (2020), 56\u201365. Issue 1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934619","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_3_2_2_63_1","unstructured":"Benjamin Wilson Judy Hoffman and Jamie Morgenstern. 2019. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097(2019).  Benjamin Wilson Judy Hoffman and Jamie Morgenstern. 2019. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097(2019)."},{"key":"e_1_3_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1073"},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415851"},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST47406.2019.8986943"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.70515"},{"key":"e_1_3_2_2_68_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang J\u00a0M","year":"2020","unstructured":"J\u00a0M Zhang , M Harman , L Ma , and Y Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE Transactions on Software Engineering( 2020 ), 1. https:\/\/doi.org\/10.1109\/TSE.2019.2962027 J\u00a0M Zhang, M Harman, L Ma, and Y Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering(2020), 1. https:\/\/doi.org\/10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1177\/0170840602235001"}],"event":{"name":"CHI '22: CHI Conference on Human Factors in Computing Systems","location":"New Orleans LA USA","acronym":"CHI '22","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491102.3502102","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3491102.3502102","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:22Z","timestamp":1750188682000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491102.3502102"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,29]]},"references-count":69,"alternative-id":["10.1145\/3491102.3502102","10.1145\/3491102"],"URL":"https:\/\/doi.org\/10.1145\/3491102.3502102","relation":{},"subject":[],"published":{"date-parts":[[2022,4,29]]}}}