{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:02:40Z","timestamp":1776110560913,"version":"3.50.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,7,13]],"date-time":"2018-07-13T00:00:00Z","timestamp":1531440000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1441331 and 1441291"],"award-info":[{"award-number":["1441331 and 1441291"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Microsoft"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2018,9,30]]},"abstract":"<jats:p>\n            Design sketching is an important skill for designers, engineers, and creative professionals, as it allows them to express their ideas and concepts in a visual medium. Being a critical and versatile skill for many different disciplines, courses on design sketching are often taught in universities. Courses today predominately rely on pen and paper; however, this traditional pedagogy is limited by the availability of human instructors, who can provide personalized feedback. Using a stylus-based intelligent tutoring system called\n            <jats:italic>SketchTivity<\/jats:italic>\n            , we aim to eventually mimic the feedback given by an instructor and assess student-drawn sketches to give students insight into areas for improvement. To provide effective feedback to users, it is important to identify what aspects of their sketches they should work on to improve their sketching ability. After consulting with several domain experts in sketching, we came up with several classes of features that could potentially differentiate expert and novice sketches. Because improvement on one metric, such as speed, may result in a decrease in another metric, such as accuracy, the creation of a single score may not mean much to the user. We attempted to create a single internal score that represents overall drawing skill so that the system can track improvement over time and found that this score correlates highly with expert rankings. We gathered over 2,000 sketches from 20 novices and four experts for analysis. We identified key metrics for quality assessment that were shown to significantly correlate with the quality of expert sketches and provide insight into providing intelligent user feedback in the future.\n          <\/jats:p>","DOI":"10.1145\/3181673","type":"journal-article","created":{"date-parts":[[2018,7,13]],"date-time":"2018-07-13T16:08:17Z","timestamp":1531498097000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["It\u2019s Not Just about Accuracy"],"prefix":"10.1145","volume":"8","author":[{"given":"Tracy","family":"Hammond","sequence":"first","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shalini Priya Ashok","family":"Kumar","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Runyon","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Josh","family":"Cherian","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Blake","family":"Williford","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Swarna","family":"Keshavabhotla","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephanie","family":"Valentine","sequence":"additional","affiliation":[{"name":"Sketch Recognition Lab, Texas A8M University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wayne","family":"Li","sequence":"additional","affiliation":[{"name":"College of Design, Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julie","family":"Linsey","sequence":"additional","affiliation":[{"name":"iDREEM Lab, Georgia Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,7,13]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/258549.258760"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2011.6142959"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1115\/DETC2011-48439"},{"key":"e_1_2_2_4_1","volume-title":"Proceedings of the 120th American Society for Engineering Education Annual Conference 8 Exposition (ASEE\u201913)","author":"Atilola Olufumilola","year":"2013"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0890060414000079"},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the 119th American Society for Engineering Education Annual Conference 8 Exposition (ASEE\u201912)","author":"Atilola Olufunmilola","year":"2012"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.3102\/00346543061002213"},{"key":"e_1_2_2_8_1","volume-title":"Revolutionizing Education with Digital Ink: The Impact of Pen and Touch Technology on Education","author":"Barreto Laura"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647415"},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the 21st International Joint Conference on Artifical Intelligence (IJCAI\u201909)","author":"Bhat Akshay","year":"2009"},{"key":"e_1_2_2_11_1","first-page":"1063","article-title":"Analysis of a random forests model","volume":"13","author":"Biau Gerard","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_2_12_1","unstructured":"Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc. San Francisco CA.   Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc. San Francisco CA."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0747-5632(99)00038-2"},{"key":"e_1_2_2_14_1","volume-title":"Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling (SBIM\u201912)","author":"Cummings Danielle","year":"2012"},{"key":"e_1_2_2_15_1","unstructured":"Karen Ernst daSilva. 2001. Drawing on experience: Connecting art and language. Primary Voices K-6 10 2 (2001) 2--9.  Karen Ernst daSilva. 2001. Drawing on experience: Connecting art and language. Primary Voices K-6 10 2 (2001) 2--9."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1384429.1384447"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753459"},{"key":"e_1_2_2_18_1","volume-title":"Sketching: Drawing Techniques for Product Designers","author":"Eissen Koos","year":"2009","edition":"12"},{"key":"e_1_2_2_19_1","volume-title":"Proceedings of the Graphics Interface 2009 Conference (GI\u201909)","author":"Eoff Brian David","year":"2009"},{"key":"e_1_2_2_20_1","volume-title":"Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI\u201911)","volume":"3","author":"Field Martin","year":"2011"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0055392"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.05-07-01688.1985"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2005.05.007"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.18260\/p.25007"},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI\u201903)","author":"Hammond Tracy","year":"2003"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/1597148.1597222"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2005.05.005"},{"key":"e_1_2_2_28_1","doi-asserted-by":"crossref","unstructured":"Tracy Hammond Drew Logsdon Brandon Paulson Joshua Johnston Joshua Peschel Aaron Wolin and Paul Taele. 2010. A sketch recognition system for recognizing free-hand course of action diagrams. In Innovative Applications of Artificial Intelligence. AAAI 1781--1786. http:\/\/www.aaai.org\/ocs\/index.php\/IAAI\/IAAI10\/paper\/view\/1581.  Tracy Hammond Drew Logsdon Brandon Paulson Joshua Johnston Joshua Peschel Aaron Wolin and Paul Taele. 2010. A sketch recognition system for recognizing free-hand course of action diagrams. In Innovative Applications of Artificial Intelligence. AAAI 1781--1786. http:\/\/www.aaai.org\/ocs\/index.php\/IAAI\/IAAI10\/paper\/view\/1581.","DOI":"10.1609\/aaai.v24i2.18812"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1754128"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2030365.2030369"},{"key":"e_1_2_2_31_1","unstructured":"Jay D. Helsel. 1979. Reading Engineering Drawings Through Conceptual Sketching (1st ed.). Glencoe\/Mcgraw--Hill Columbus OH USA.  Jay D. Helsel. 1979. Reading Engineering Drawings Through Conceptual Sketching (1st ed.). Glencoe\/Mcgraw--Hill Columbus OH USA."},{"key":"e_1_2_2_32_1","unstructured":"Ethan Hilton Wayne Li Tracy Hammond and Julie Linsey. 2018. Effectively teaching sketching in engineering curricula. (unpublished).  Ethan Hilton Wayne Li Tracy Hammond and Julie Linsey. 2018. Effectively teaching sketching in engineering curricula. (unpublished)."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1115\/DETC2016-60250"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.18260\/1-2--27874"},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of the International Conference on Engineering Design (ICED\u201917)","volume":"224","author":"Hilton Ethan","year":"2017"},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of the 4th International Conference on Design Creativity. The Design Society, Article 43","author":"Hilton Ethan","year":"2016"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2501997"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/274644.274672"},{"key":"e_1_2_2_39_1","first-page":"174","article-title":"The correlation of visual memory and perception of perspective with drawing ability","volume":"15","author":"Jones Elmer E.","year":"1922","journal-title":"School Soc."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186562.1015741"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2005.05.004"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092907.3092911"},{"key":"e_1_2_2_43_1","volume-title":"Proceedings of the 2014 Intelligent User Interfaces Workshop on Sketch Recognition. ACM","author":"Kim Hong-Hoe","year":"2014"},{"key":"e_1_2_2_44_1","volume-title":"The Impact of Pen and Touch Technology on Education","author":"Kim Hong-Hoe","edition":"1"},{"key":"e_1_2_2_45_1","unstructured":"Joseph A. Koncelik and Kevin Reeder. 2008. Conceptual Drawing (1 ed.). Course Technology Boston MA.  Joseph A. Koncelik and Kevin Reeder. 2008. Conceptual Drawing (1 ed.). Course Technology Boston MA."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964922"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/EVA2016.28"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2013.2250282"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645643"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2212776.2223730"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2012.2190389"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2010.2086444"},{"key":"e_1_2_2_53_1","doi-asserted-by":"crossref","unstructured":"David E. Meyer J. E. Keith-Smith Sylvan Kornblum Richard A. Abrams and Charles E. Wright. 1990. Speed-accuracy tradeoffs in aimed movements: Toward a theory of rapid voluntary action. In Attention and Performance 13: Motor Representation and Control. Lawrence Erlbaum Associates Inc. Hillsdale NJ 173--226.  David E. Meyer J. E. Keith-Smith Sylvan Kornblum Richard A. Abrams and Charles E. Wright. 1990. Speed-accuracy tradeoffs in aimed movements: Toward a theory of rapid voluntary action. In Attention and Performance 13: Motor Representation and Control. Lawrence Erlbaum Associates Inc. Hillsdale NJ 173--226.","DOI":"10.4324\/9780203772010-6"},{"key":"e_1_2_2_54_1","volume-title":"Proceedings of the 2000 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 464--471","author":"Miller Erik G."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2732158.2732194"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2008.09.013"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1476-8070.2005.00439.x"},{"key":"e_1_2_2_58_1","volume-title":"Proceedings of the ACM Symposium on User Interface Software and Technology (UIST\u201907)","author":"Paulson Brandon","year":"2007"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378773.1378775"},{"key":"e_1_2_2_60_1","volume-title":"Proceedings of the HCC Workshop: Sketch Tools for Diagramming. VL\/HCC, Herrsching am Ammersee, Germany, 57--63","author":"Paulson Brandon","year":"2008"},{"key":"e_1_2_2_61_1","unstructured":"Dan Roam. 2013. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures (1 ed.). Portfolio Publishing London England United Kingdom.  Dan Roam. 2013. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures (1 ed.). Portfolio Publishing London England United Kingdom."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/122718.122753"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1080\/00393541.2008.11518756"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281500.1281547"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1185657.1185783"},{"key":"e_1_2_2_66_1","doi-asserted-by":"crossref","unstructured":"Herbert A. Simon. 1983. Why should machines learn? In Machine Learning Ryszard S. Michalski Jaime G. Carbonell and Tom M. Mitchell (Eds.). Springer Berlin 25--37.  Herbert A. Simon. 1983. Why should machines learn? In Machine Learning Ryszard S. Michalski Jaime G. Carbonell and Tom M. Mitchell (Eds.). Springer Berlin 25--37.","DOI":"10.1016\/B978-0-08-051054-5.50006-6"},{"key":"e_1_2_2_67_1","volume-title":"Learning Curves: An Inspiring Guide to Improve Your Design Sketch Skills","author":"Sj\u00f6l\u00e9n Klara","year":"2011"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1080\/09500690802595839"},{"key":"e_1_2_2_69_1","volume-title":"Proceedings of the 21st Innovative Applications of Artificial Intelligence Conference (IAAI\u201915)","author":"Taele Paul","year":"2009"},{"key":"e_1_2_2_70_1","volume-title":"Proceedings of the 27th Innovative Applications of Artificial Intelligence Conference (IAAI\u201915)","author":"Taele Paul","year":"2015"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvlc.2009.12.004"},{"key":"e_1_2_2_72_1","first-page":"91","article-title":"Mechanix: A sketch-based tutoring system that automatically corrects hand-sketched statics homework. In The Impact of Pen and Touch Technology on Education, A. Adler T. Hammond, S. Valentine and M. Payton (Eds.). Springer, Switzerland","volume":"9","author":"Valentine Stephanie","year":"2015","journal-title":"Chapter"},{"key":"e_1_2_2_73_1","volume-title":"Proceedings of the 24th Innovative Applications of Artificial Intelligence Conference (IAAI\u201912)","author":"Valentine Stephanie","year":"2012"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v34i1.2437"},{"key":"e_1_2_2_75_1","unstructured":"Antony White. 1990. Harrap\u2019s Illustrated Dictionary of Art 8 Artists. Harrap\u2019s Reference Edin.  Antony White. 1990. Harrap\u2019s Illustrated Dictionary of Art 8 Artists. Harrap\u2019s Reference Edin."},{"key":"e_1_2_2_76_1","volume-title":"Extended Abstract for Workshop SketchingDIS: Hand-drawn Sketching in HCI Workshop at Designing Interactive Systems Conference (DIS\u201917)","author":"Williford Blake","year":"2017"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130859.3130861"},{"key":"e_1_2_2_78_1","volume-title":"Revolutionizing Education with Digital Ink: The Impact of Pen and Touch Technology on Education","author":"Williford Blake"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294211.1294238"},{"key":"e_1_2_2_80_1","volume-title":"Proceedings of the Fifth Eurographics Conference on Sketch-Based Interfaces and Modeling (SBM\u201908)","author":"Wolin Aaron","year":"2008"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647399"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3181673","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3181673","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3181673","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:37:49Z","timestamp":1750282669000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3181673"}},"subtitle":["Metrics That Matter When Modeling Expert Sketching Ability"],"short-title":[],"issued":{"date-parts":[[2018,7,13]]},"references-count":81,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9,30]]}},"alternative-id":["10.1145\/3181673"],"URL":"https:\/\/doi.org\/10.1145\/3181673","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,13]]},"assertion":[{"value":"2016-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}