{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:40:58Z","timestamp":1775230858019,"version":"3.50.1"},"reference-count":87,"publisher":"Informa UK Limited","issue":"5-6","content-domain":{"domain":["www.tandfonline.com"],"crossmark-restriction":true},"short-container-title":["Human\u2013Computer Interaction"],"published-print":{"date-parts":[[2020,11]]},"DOI":"10.1080\/07370024.2020.1734931","type":"journal-article","created":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T11:39:58Z","timestamp":1588246798000},"page":"413-451","update-policy":"https:\/\/doi.org\/10.1080\/tandf_crossmark_01","source":"Crossref","is-referenced-by-count":87,"title":["Interactive machine teaching: a human-centered approach to building machine-learned models"],"prefix":"10.1080","volume":"35","author":[{"given":"Gonzalo","family":"Ramos","sequence":"first","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Christopher","family":"Meek","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Patrice","family":"Simard","sequence":"additional","affiliation":[{"name":"Microsoft"}]},{"given":"Jina","family":"Suh","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Soroush","family":"Ghorashi","sequence":"additional","affiliation":[{"name":"Microsoft"}]}],"member":"301","published-online":{"date-parts":[[2020,4,30]]},"reference":[{"key":"e_1_3_2_2_1","unstructured":"Amazon. (2020). Amazon Sagemaker: Machine learning for every developer and data scientist . Retrieved from https:\/\/aws.amazon.com\/sagemaker"},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_6_1","volume-title":"Behaviorism, neobehaviorism, and cognitivism in learning theory: Historical and contemporary perspectives","author":"Amsel A.","year":"1989","unstructured":"Amsel, A. (1989). Behaviorism, neobehaviorism, and cognitivism in learning theory: Historical and contemporary perspectives. Hillsdale, New Jersey: Lawrence Erlbaum."},{"key":"e_1_3_2_7_1","unstructured":"Apple. (2020). Turi create simplifies the development of custom machine learning models . Retrieved from https:\/\/github.com\/apple\/turicreate"},{"issue":"1","key":"e_1_3_2_8_1","first-page":"1737","article-title":"Random forests, decision trees, and categorical predictors: The absent levels problem","volume":"19","author":"Au T. C.","year":"2018","unstructured":"Au, T. C. (2018). Random forests, decision trees, and categorical predictors: The absent levels problem. The Journal of Machine Learning Research, 19(1), 1737\u20131766.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_2_9_1","unstructured":"Azure M. (2020). Azure ML Studio . Retrieved from https:\/\/studio.azureml.net\/"},{"key":"e_1_3_2_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/()1097-4571"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/273133.274308"},{"key":"e_1_3_2_12_1","unstructured":"Bonsai. (2020a). Bonsai: BRAINs for autonomous systems . Retrieved from https:\/\/bons.ai"},{"key":"e_1_3_2_13_1","unstructured":"Bonsai. (2020b). Inkling: Bonsai\u2019s special-purpose programming language for training AI . Retrieved from https:\/\/docs.bons.ai\/guides\/inkling2-guide.html"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/800027.808439"},{"key":"e_1_3_2_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2014.08.005"},{"key":"e_1_3_2_16_1","unstructured":"Card S. K. Mackinlay J. D. & Shneiderman B. (1999). Readings in Information Visualization: Using Vision to Think. Information Visualization 1\u201334. San Francisco CA: Morgan Kaufmann Publishers Inc."},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788613"},{"key":"e_1_3_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702146"},{"key":"e_1_3_2_19_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00568ED1V01Y201402AIM028"},{"key":"e_1_3_2_20_1","doi-asserted-by":"publisher","DOI":"10.3115\/1613984.1614012"},{"key":"e_1_3_2_21_1","unstructured":"Doshi-Velez F. & Kim B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 ."},{"issue":"2","key":"e_1_3_2_23_1","first-page":"8","article-title":"A review of user interface design for interactive machine learning","volume":"8","author":"Dudley J. J.","year":"2018","unstructured":"Dudley, J. J., & Kristensson, P. O. (2018). A review of user interface design for interactive machine learning. ACM Transactions on Interactive Intelligent Systems (Tiis), 8(2), 8.","journal-title":"ACM Transactions on Interactive Intelligent Systems (Tiis)"},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_3_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242596"},{"key":"e_1_3_2_26_1","volume-title":"Proceedings of the ISMIR 2010 international society for music information retrieval conference, Utrecht, the Netherlands","author":"Fiebrink R.","year":"2010","unstructured":"Fiebrink, R., & Cook, P. R. (2010). The wekinator: A system for real-time, interactive machine learning in music. Proceedings of the ISMIR 2010 international society for music information retrieval conference, Utrecht, the Netherlands."},{"key":"e_1_3_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357061"},{"key":"e_1_3_2_28_1","first-page":"3348","volume-title":"Advances in neural information processing systems 31, San Jose, CA","author":"Fusi N.","year":"2018","unstructured":"Fusi, N., Sheth, R., & Elibol, M. (2018). Probabilistic matrix factorization for automated machine learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in neural information processing systems 31, San Jose, CA (pp. 3348\u20133357). Curran Associates, Inc."},{"key":"e_1_3_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_30_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v38i3.2741"},{"key":"e_1_3_2_31_1","unstructured":"Google. (2020a). AI platform: Create your AI applications once then run them easily on both GCP and on-premises . Retrieved from https:\/\/cloud.google.com\/ai-platform"},{"key":"e_1_3_2_32_1","unstructured":"Google. (2020b). Teachable machine: Train a computer to recognize your own images sounds & poses . Retrieved from https:\/\/teachablemachine.withgoogle.com\/"},{"key":"e_1_3_2_33_1","unstructured":"Google. (2020c). Tensorflow: An end-to-end open source machine learning platform . Retrieved from https:\/\/www.tensorflow.org\/"},{"key":"e_1_3_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300809"},{"key":"e_1_3_2_35_1","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/302979.303030","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems","author":"Horvitz E.","year":"1999","unstructured":"Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of the SIGCHI conference on human factors in computing systems (pp. 159\u2013166). New York, NY: ACM."},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13218-015-0381-0"},{"key":"e_1_3_2_37_1","unstructured":"IBM. (2020). Watson studio: Simplify and scale data science to predict and optimize your business outcomes . Retrieved from https:\/\/www.ibm.com\/cloud\/watson-studio"},{"key":"e_1_3_2_38_1","unstructured":"Jandot C. Simard P. Chickering M. Grangier D. & Suh J. (2016). Interactive semantic featuring for text classification . Retrieved from https:\/\/arxiv.org\/abs\/1606.07545v1"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298877"},{"key":"e_1_3_2_40_1","unstructured":"Jupyter P. (2020). Jupyter notebooks . Retrieved from https:\/\/jupyter.org\/"},{"key":"e_1_3_2_41_1","unstructured":"Kane D. M. Lovett S. Moran S. & Zhang J. (2017). Active classification with comparison queries. CoRR abs\/1704.03564 . Retrieved from http:\/\/arxiv.org\/abs\/1704.03564"},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2015.7344858"},{"issue":"2","key":"e_1_3_2_44_1","article-title":"Interaction algorithm effect on human experience with reinforcement learning","volume":"7","author":"Krening S.","year":"2018","unstructured":"Krening, S., & Feigh, K. M. (2018, October). Interaction algorithm effect on human experience with reinforcement learning. Journal of Human-Robot Interaction, 7(2). https:\/\/dl.acm.org\/doi\/10.1145\/3277904","journal-title":"Journal of Human-Robot Interaction"},{"key":"e_1_3_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3322276.3322379"},{"key":"e_1_3_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557238"},{"key":"e_1_3_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_3_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_3_2_49_1","unstructured":"Lane D. (2020). Machine learning for kids . Retrieved from https:\/\/machinelearningforkids.co.uk"},{"key":"e_1_3_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_51_1","first-page":"4765","volume-title":"Advances in neural information processing systems 30, San Jose, CA","author":"Lundberg S. M.","year":"2017","unstructured":"Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems 30, San Jose, CA (pp. 4765\u20134774). Curran Associates, Inc."},{"key":"e_1_3_2_52_1","unstructured":"Matas M. Menges A. & Beissinger M. (2020). Lobe: Deep learning made simple . Retrieved from https:\/\/lobe.ai\/"},{"key":"e_1_3_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169215"},{"key":"e_1_3_2_54_1","unstructured":"Meek C. (2016). A characterization of prediction errors. CoRR abs\/1611.05955 . Retrieved from http:\/\/arxiv.org\/abs\/1611.05955"},{"key":"e_1_3_2_55_1","unstructured":"Meek C. Simard P. & Zhu X. (2016). Analysis of a design pattern for teaching with features and labels. CoRR abs\/1611.05950 . Retrieved from http:\/\/arxiv.org\/abs\/1611.05950"},{"key":"e_1_3_2_56_1","unstructured":"Microsoft. (2020). LUIS: Language understanding . Retrieved from https:\/\/www.luis.ai\/"},{"key":"e_1_3_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287574"},{"issue":"2","key":"e_1_3_2_58_1","first-page":"250","article-title":"Applying radical constructivism to machine learning: A pilot study in assistive robotics","volume":"13","author":"Nowak M.","year":"2018","unstructured":"Nowak, M., Castellini, C., & Massironi, C. (2018). Applying radical constructivism to machine learning: A pilot study in assistive robotics. Constructivist Foundations, 13(2), 250\u2013262.","journal-title":"Constructivist Foundations"},{"key":"e_1_3_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2908812.2908918"},{"key":"e_1_3_2_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-31204-0_9"},{"key":"e_1_3_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866038"},{"key":"e_1_3_2_62_1","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa F.","year":"2011","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., \u2026 Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825\u20132830.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2007.53"},{"key":"e_1_3_2_64_1","volume-title":"Perspectives on learning","author":"Phillips D.","year":"2015","unstructured":"Phillips, D., & Soltis, J. (2015). Perspectives on learning (5th ed.). New York: Teachers College Press.","edition":"5"},{"key":"e_1_3_2_65_1","volume-title":"Proceedings of the 2005 international conference on intelligence analysis","volume":"5","author":"Pirolli P.","year":"2005","unstructured":"Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of the 2005 international conference on intelligence analysis (Vol. 5), McLean, VA."},{"key":"e_1_3_2_66_1","unstructured":"Prodigy. (2020). Prodigy: Radically efficient machine teaching . Retrieved from https:\/\/prodi.gy\/"},{"key":"e_1_3_2_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00552-1"},{"key":"e_1_3_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1592761.1592779"},{"key":"e_1_3_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_70_1","first-page":"167","volume-title":"Proceedings of the IUI 2009 international conference on intelligent user interfaces","author":"Ritter A.","year":"2009","unstructured":"Ritter, A., & Basu, S. (2009). Learning to generalize for complex selection tasks. Proceedings of the IUI 2009 international conference on intelligent user interfaces (pp. 167\u2013176). New York, NY: ACM."},{"key":"e_1_3_2_71_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_3_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892547"},{"key":"e_1_3_2_73_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000046"},{"key":"e_1_3_2_74_1","first-page":"2503","volume-title":"Advances in neural information processing systems 28, Montreal, Canada","author":"Sculley D.","year":"2015","unstructured":"Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., \u2026 Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in neural information processing systems 28, Montreal, Canada (pp. 2503\u20132511). Curran Associates, Inc."},{"key":"e_1_3_2_75_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-01560-1"},{"key":"e_1_3_2_76_1","unstructured":"Simard P. Y. Amershi S. Chickering D. M. Pelton A. E. Ghorashi S. Meek C. Ramos G. Suh J. Verwey J. Wang M. Wernsing J.\u00a0(2017). Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742 ."},{"key":"e_1_3_2_77_1","unstructured":"Spacy. (2020). Spacy: Industrial-strength natural language processing . Retrieved from https:\/\/spacy.io\/"},{"key":"e_1_3_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3241379"},{"key":"e_1_3_2_79_1","volume-title":"Reinforcement learning: An introduction","author":"Sutton R.","year":"2018","unstructured":"Sutton, R., & Barto, A. (2018). Reinforcement learning: An introduction. Cambridge, MA: MIT Press."},{"key":"e_1_3_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098161"},{"key":"e_1_3_2_81_1","doi-asserted-by":"publisher","DOI":"10.1038\/srep37854"},{"key":"e_1_3_2_82_1","volume-title":"Piaget\u2019s theory of cognitive and affective development: Foundations of constructivism","author":"Wadsworth B.","year":"1996","unstructured":"Wadsworth, B. (1996). Piaget\u2019s theory of cognitive and affective development: Foundations of constructivism. London, UK: Longman."},{"key":"e_1_3_2_83_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29387-1_34"},{"key":"e_1_3_2_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2110363.2110464"},{"key":"e_1_3_2_85_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2001.0499"},{"key":"e_1_3_2_86_1","unstructured":"Webhose. (2020). Webhose.io: Tap into web content at scale . Retrieved from http:\/\/webhose.io"},{"key":"e_1_3_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3282486"},{"key":"e_1_3_2_88_1","first-page":"2431","volume-title":"Advances in neural information processing systems 30","author":"Xu Y.","year":"2017","unstructured":"Xu, Y., Zhang, H., Miller, K., Singh, A., & Dubrawski, A. (2017). Noise-tolerant interactive learning using pairwise comparisons. Advances in neural information processing systems 30 (pp. 2431\u20132440). Curran Associates, Inc."},{"key":"e_1_3_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196729"},{"key":"e_1_3_2_90_1","volume-title":"Proceedings of the AAAI 2015 conference on artificial intelligence","author":"Zhu X.","year":"2015","unstructured":"Zhu, X. (2015). Machine teaching: An inverse problem to machine learning and an approach toward optimal education. Proceedings of the AAAI 2015 conference on artificial intelligence."},{"key":"e_1_3_2_91_1","unstructured":"Zhu X. Singla A. Zilles S. & Rafferty A. N. (2018). An overview of machine teaching. CoRR abs\/1801.05927 . Retrieved from http:\/\/arxiv.org\/abs\/1801.05927"}],"container-title":["Human\u2013Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.tandfonline.com\/doi\/pdf\/10.1080\/07370024.2020.1734931","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,2]],"date-time":"2023-11-02T09:33:36Z","timestamp":1698917616000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.tandfonline.com\/doi\/full\/10.1080\/07370024.2020.1734931"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,30]]},"references-count":87,"journal-issue":{"issue":"5-6","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["10.1080\/07370024.2020.1734931"],"URL":"https:\/\/doi.org\/10.1080\/07370024.2020.1734931","relation":{},"ISSN":["0737-0024","1532-7051"],"issn-type":[{"value":"0737-0024","type":"print"},{"value":"1532-7051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,30]]},"assertion":[{"value":"The publishing and review policy for this title is described in its Aims & Scope.","order":1,"name":"peerreview_statement","label":"Peer Review Statement"},{"value":"http:\/\/www.tandfonline.com\/action\/journalInformation?show=aimsScope&journalCode=hhci20","URL":"http:\/\/www.tandfonline.com\/action\/journalInformation?show=aimsScope&journalCode=hhci20","order":2,"name":"aims_and_scope_url","label":"Aim & Scope"},{"value":"2019-06-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-02-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-04-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}