{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:41:00Z","timestamp":1776112860717,"version":"3.50.1"},"reference-count":106,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2019,11,29]],"date-time":"2019-11-29T00:00:00Z","timestamp":1574985600000},"content-version":"unspecified","delay-in-days":28,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AIEDAM"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Sketching is a natural and intuitive communication tool used for expressing concepts or ideas which are difficult to communicate through text or speech alone. Sketching is therefore used for a variety of purposes, from the expression of ideas on two-dimensional (2D) physical media, to object creation, manipulation, or deformation in three-dimensional (3D) immersive environments. This variety in sketching activities brings about a range of technologies which, while having similar scope, namely that of recording and interpreting the sketch gesture to effect some interaction, adopt different interpretation approaches according to the environment in which the sketch is drawn. In fields such as product design, sketches are drawn at various stages of the design process, and therefore, designers would benefit from sketch interpretation technologies which support these differing interactions. However, research typically focuses on one aspect of sketch interpretation and modeling such that literature on available technologies is fragmented and dispersed. In this paper, we bring together the relevant literature describing technologies which can support the product design industry, namely technologies which support the interpretation of sketches drawn on 2D media, sketch-based search interactions, as well as sketch gestures drawn in 3D media. This paper, therefore, gives a holistic view of the algorithmic support that can be provided in the design process. In so doing, we highlight the research gaps and future research directions required to provide full sketch-based interaction support.<\/jats:p>","DOI":"10.1017\/s0890060419000349","type":"journal-article","created":{"date-parts":[[2019,11,29]],"date-time":"2019-11-29T05:31:01Z","timestamp":1575005461000},"page":"370-388","source":"Crossref","is-referenced-by-count":27,"title":["Sketch-based interaction and modeling: where do we stand?"],"prefix":"10.1017","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6580-3424","authenticated-orcid":false,"given":"Alexandra","family":"Bonnici","sequence":"first","affiliation":[]},{"given":"Alican","family":"Akman","sequence":"additional","affiliation":[]},{"given":"Gabriel","family":"Calleja","sequence":"additional","affiliation":[]},{"given":"Kenneth P.","family":"Camilleri","sequence":"additional","affiliation":[]},{"given":"Patrick","family":"Fehling","sequence":"additional","affiliation":[]},{"given":"Alfredo","family":"Ferreira","sequence":"additional","affiliation":[]},{"given":"Florian","family":"Hermuth","sequence":"additional","affiliation":[]},{"given":"Johann Habakuk","family":"Israel","sequence":"additional","affiliation":[]},{"given":"Tom","family":"Landwehr","sequence":"additional","affiliation":[]},{"given":"Juncheng","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Natasha M. J.","family":"Padfield","sequence":"additional","affiliation":[]},{"given":"T. Metin","family":"Sezgin","sequence":"additional","affiliation":[]},{"given":"Paul L.","family":"Rosin","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2019,11,29]]},"reference":[{"key":"S0890060419000349_ref20","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/HCI2009.8"},{"key":"S0890060419000349_ref30","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925946"},{"key":"S0890060419000349_ref79","doi-asserted-by":"publisher","DOI":"10.1109\/34.993554"},{"key":"S0890060419000349_ref90","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174218"},{"key":"S0890060419000349_ref55","doi-asserted-by":"publisher","DOI":"10.1145\/3103010.3103021"},{"key":"S0890060419000349_ref33","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13334"},{"key":"S0890060419000349_ref5","doi-asserted-by":"publisher","DOI":"10.1145\/3202661"},{"key":"S0890060419000349_ref99","unstructured":"Wiese E , Israel JH , Meyer A and Bongartz S (2010) Investigating the learnability of immersive free-hand sketching. Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium, Annecy, France, June 07\u201310, 2010. Eurographics Association, pp. 135\u2013142."},{"key":"S0890060419000349_ref46","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.1060"},{"key":"S0890060419000349_ref1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-0789-7"},{"key":"S0890060419000349_ref38","unstructured":"Igarashi T , Matsuoka S and Tanaka H (1999) Teddy: a sketching interface for 3D freeform design. Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, August 8\u201313, 1999, Los Angeles, California, USA. ACM Press\/Addison-Wesley Publishing Co, pp. 409\u2013416."},{"key":"S0890060419000349_ref65","doi-asserted-by":"publisher","DOI":"10.1207\/S15326950DP3303_4"},{"key":"S0890060419000349_ref89","doi-asserted-by":"publisher","DOI":"10.1109\/34.659930"},{"key":"S0890060419000349_ref99a","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.385"},{"key":"S0890060419000349_ref7","unstructured":"Bonnici A , Bugeja D and Azzopardi G (2018) Vectorisation of sketches with shadows and shading using COSFIRE filters. Proceedings of the ACM Symposium on Document Engineering, Halifax, NS, Canada, August 28\u201331, 2018, pp. 1\u201310."},{"key":"S0890060419000349_ref71","unstructured":"Olsen L , Samavati F , Sousa MC and Jorge J (2008) A Taxonomy of Modelling Techniques Using Sketch-Based Interfaces. Eurographics state of the art reports, 1 (1.4), 1."},{"key":"S0890060419000349_ref94","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1093\/acprof:oso\/9780195381634.003.0004","volume-title":"Tools for Innovation: The Science behind the Practical Methods That Drive New Ideas","author":"Tversky","year":"2009"},{"key":"S0890060419000349_ref73","doi-asserted-by":"publisher","DOI":"10.1207\/S15327051HCI1504_1"},{"key":"S0890060419000349_ref17","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531334"},{"key":"S0890060419000349_ref58","unstructured":"Lu C , Xu L and Jia J (2012) Combining sketch and tone for pencil drawing production. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering, Annecy, France, June 04\u201306, 2012. Germany: Eurographics Association, pp. 65\u201373."},{"key":"S0890060419000349_ref66","unstructured":"Munehiro N and Huang TS (2001) An interactive 3D visualization for content-based image retrieval. Proceedings of the IEEE International Conference on Multimedia, 22\u201325 August 2001, Tokyo, Japan, Japan, pp. 22\u201325."},{"key":"S0890060419000349_ref44","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026183017197"},{"key":"S0890060419000349_ref62","doi-asserted-by":"publisher","DOI":"10.1145\/1281500.1281542"},{"key":"S0890060419000349_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-64381-8_65"},{"key":"S0890060419000349_ref18","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2009.05.004"},{"key":"S0890060419000349_ref88","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2002.1023802"},{"key":"S0890060419000349_ref35","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2006.127"},{"key":"S0890060419000349_ref76","first-page":"48","volume-title":"Lecture Notes in Computer Science, Graphics Recognition \u2013 Algorithms and Applications","volume":"1389","author":"Ramel","year":"1998"},{"key":"S0890060419000349_ref95","unstructured":"Varley P (2009) The use of neighbourhood matching in constructing hidden object topology. Proceedings of the World Congress on Engineering, July 1\u20133, 2009, London, U.K., Vol. 1."},{"key":"S0890060419000349_ref85","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925972"},{"key":"S0890060419000349_ref11","first-page":"1","article-title":"Vectorization of line drawing image based on junction analysis","volume":"58","author":"Chen","year":"2015","journal-title":"Science China Information Sciences"},{"key":"S0890060419000349_ref42","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276401"},{"key":"S0890060419000349_ref87","doi-asserted-by":"crossref","first-page":"1214","DOI":"10.1109\/TMI.2006.879967","article-title":"Retinal vessel segmentation using the 2-D Gabor wavelet and supervised classification","volume":"25","author":"Soares","year":"2006","journal-title":"IEEE Transactions on Medical Imaging"},{"key":"S0890060419000349_ref61","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2010.2064333"},{"key":"S0890060419000349_ref68","unstructured":"Nguyen-Phuoc TH , Li C , Balaban S and Yang Y (2018) RenderNet: a deep convolutional network for differentiable rendering from 3D shapes. Advances in Neural Information Processing Systems, Montr\u00e9al, Canada, December 03\u201308, 2018, pp. 7891\u20137901."},{"key":"S0890060419000349_ref67","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276429"},{"key":"S0890060419000349_ref49","unstructured":"Kirk D , Rodden T and Stanton Fraser D (2007) Turn it this way: grounding collaborative action with remote gestures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, California, USA, April 28\u2013May 03, 2007. New York: ACM, pp. 1039\u20131048."},{"key":"S0890060419000349_ref2","doi-asserted-by":"publisher","DOI":"10.1145\/1384429.1384449"},{"key":"S0890060419000349_ref6","unstructured":"Bonnici A , Israel JH , Muscat AM , Camilleri D , Camilleri K and Rothenburg U (2015) Investigating user response to a hybrid sketch-based interface for creating 3D virtual models in an immersive environment. Proceedings of the 10th International Conference on Computer Graphics Theory and Applications, Berlin, Germany, March 11\u201314, 2015. Berlin: SCITEPRESS, pp. 470\u2013477."},{"key":"S0890060419000349_ref8","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.1986.4767851"},{"key":"S0890060419000349_ref9","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1109\/TPAMI.2007.1185","article-title":"What the back of the object looks like: 3D reconstruction from line drawings without hidden lines","volume":"30","author":"Cao","year":"2008","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"S0890060419000349_ref10","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8659.00669"},{"key":"S0890060419000349_ref34","first-page":"69","volume-title":"Visual and Spatial Reasoning in Design III","author":"Heiser","year":"2004"},{"key":"S0890060419000349_ref12","doi-asserted-by":"crossref","unstructured":"Cherubini M , N\u00fcssli MA and Dillenbourg P (2008) Deixis and gaze in collaborative work at a distance (over a shared map): a computational model to detect misunderstandings. Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, Savannah, Georgia, March 26\u201328, 2008. New York: ACM.","DOI":"10.1145\/1344471.1344515"},{"key":"S0890060419000349_ref15","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(71)90005-1"},{"key":"S0890060419000349_ref16","doi-asserted-by":"publisher","DOI":"10.1145\/1360612.1360687"},{"key":"S0890060419000349_ref100","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601128"},{"key":"S0890060419000349_ref19","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.70835"},{"key":"S0890060419000349_ref22","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173923"},{"key":"S0890060419000349_ref24","doi-asserted-by":"publisher","DOI":"10.1145\/3203197"},{"key":"S0890060419000349_ref25","doi-asserted-by":"publisher","DOI":"10.1109\/34.754586"},{"key":"S0890060419000349_ref26","unstructured":"Drascic D and Milgram P (1996) Perceptual issues in augmented reality. SPIE Stereoscopic Displays and Virtual Reality Systems III, San Jose, CA, United States, Vol. 2653, pp. 123\u2013135."},{"key":"S0890060419000349_ref84","doi-asserted-by":"publisher","DOI":"10.1109\/SMI.2004.1314504"},{"key":"S0890060419000349_ref51","doi-asserted-by":"publisher","DOI":"10.1145\/223904.223910"},{"key":"S0890060419000349_ref27","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2450"},{"key":"S0890060419000349_ref97","first-page":"19","volume-title":"The Psychology of Computer Vision","author":"Waltz","year":"1975"},{"key":"S0890060419000349_ref28","volume-title":"Sketching. Drawing Techniques for Product Designers","author":"Eissen","year":"2007"},{"key":"S0890060419000349_ref29","first-page":"1","article-title":"Sketch-based shape retrieval","volume":"31","author":"Eitz","year":"2012","journal-title":"ACM Transactions on Graphics"},{"key":"S0890060419000349_ref32","unstructured":"Fonseca MJ , Pimentel C and Jorge JA (2002) CALI: an online scribble recognizer for calligraphic interfaces. Sketch Understanding, Papers from the 2002 AAAI Spring Symposium, 25\u201327 March 2002, Stanford, California."},{"key":"S0890060419000349_ref36","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.243"},{"key":"S0890060419000349_ref3","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025474"},{"key":"S0890060419000349_ref37","first-page":"295","article-title":"Impossible objects as nonsense sentences","volume":"6","author":"Huffman","year":"1971","journal-title":"Machine Intelligence"},{"key":"S0890060419000349_ref39","doi-asserted-by":"publisher","DOI":"10.1109\/LGRS.2015.2458898"},{"key":"S0890060419000349_ref41","doi-asserted-by":"publisher","DOI":"10.1109\/VRAIS.1993.380767"},{"key":"S0890060419000349_ref43","unstructured":"Kang H , Lee S and Chui CK (2007) Coherent line drawing. Proceedings of the 5th International Symposium on Non-Photorealistic Animation and Rendering, San Diego, California, August 04\u201305, 2007. New York: ACM, pp. 43\u201350."},{"key":"S0890060419000349_ref45","doi-asserted-by":"crossref","unstructured":"Keefe DF , Feliz DA , Moscovich T , Laidlaw DH and LaViola JJ Jr (2001) Cavepainting: a fully immersive 3D artistic medium and interactive experience. Proceedings of the 2001 Symposium on Interactive 3D Graphics. New York: ACM, pp. 85\u201393.","DOI":"10.1145\/364338.364370"},{"key":"S0890060419000349_ref47","doi-asserted-by":"publisher","DOI":"10.1007\/11767978_34"},{"key":"S0890060419000349_ref48","doi-asserted-by":"publisher","DOI":"10.1016\/j.cad.2010.03.002"},{"key":"S0890060419000349_ref23","doi-asserted-by":"publisher","DOI":"10.1145\/882262.882354"},{"key":"S0890060419000349_ref50","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2009.03.001"},{"key":"S0890060419000349_ref52","doi-asserted-by":"publisher","DOI":"10.3390\/jimaging5020026"},{"key":"S0890060419000349_ref53","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073675"},{"key":"S0890060419000349_ref54","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275051"},{"key":"S0890060419000349_ref56","doi-asserted-by":"publisher","DOI":"10.1145\/1281500.1281555"},{"key":"S0890060419000349_ref57","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/TPAMI.2010.49","article-title":"Decomposition of complex line drawings with hidden lines for 3D planar-faced manifold object reconstruction","volume":"33","author":"Liu","year":"2011","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"S0890060419000349_ref60","doi-asserted-by":"publisher","DOI":"10.1109\/3DV.2017.00018"},{"key":"S0890060419000349_ref59","doi-asserted-by":"crossref","unstructured":"Lugt RVD (2002) Functions of sketching in design idea generation meetings. Proceedings of the 4th Conference on Creativity & Cognition. Loughborough, UK: ACM, pp. 72\u201379.","DOI":"10.1145\/581710.581723"},{"key":"S0890060419000349_ref63","unstructured":"Massie TH and Salisbury JK (1994) The PHANToM haptic interface: a device for probing virtual objects. Proceedings of the ASME Winter Annual Meeting, Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, November 1994, Chicago, USA, Vol. 55(1), pp. 295\u2013300."},{"key":"S0890060419000349_ref69","doi-asserted-by":"publisher","DOI":"10.1145\/2421636.2421640"},{"key":"S0890060419000349_ref74","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2013.06.027"},{"key":"S0890060419000349_ref70","doi-asserted-by":"publisher","DOI":"10.1145\/1744161.1744163"},{"key":"S0890060419000349_ref72","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2011.84"},{"key":"S0890060419000349_ref75","doi-asserted-by":"publisher","DOI":"10.1007\/11428848_45"},{"key":"S0890060419000349_ref77","doi-asserted-by":"crossref","unstructured":"Ren JS and Xu L (2015) On vectorization of deep convolutional neural networks for vision tasks. Proceedings of the 29th AAAI Conference on Artificial Intelligence, January 25\u201330, 2015, Austin Texas, USA.","DOI":"10.1609\/aaai.v29i1.9488"},{"key":"S0890060419000349_ref78","first-page":"234","volume-title":"Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science","volume":"9351","author":"Ronneberger","year":"2015"},{"key":"S0890060419000349_ref80","doi-asserted-by":"publisher","DOI":"10.1145\/365024.365114"},{"key":"S0890060419000349_ref81","doi-asserted-by":"crossref","unstructured":"Schl\u00f6sser C (2018) Towards concise gaze sharing. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (ETRA \u201918), Warsaw, Poland, June 14\u201317, 2018. New York, NY, USA: ACM, Article 78, p. 3, doi:10.1145\/3204493.3207416.","DOI":"10.1145\/3204493.3207416"},{"key":"S0890060419000349_ref83","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/S0926-5805(99)00052-7","article-title":"Digital clay: deriving digital models from freehand sketches","volume":"9","author":"Schweikardt","year":"2000","journal-title":"Automation in Construction"},{"key":"S0890060419000349_ref21","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2817042"},{"key":"S0890060419000349_ref86","doi-asserted-by":"publisher","DOI":"10.1145\/2503512.2503539"},{"key":"S0890060419000349_ref91","doi-asserted-by":"publisher","DOI":"10.1016\/j.cad.2005.02.010"},{"key":"S0890060419000349_ref92","volume-title":"Computer Vision: Algorithms and Applications","author":"Szeliski","year":"2010"},{"key":"S0890060419000349_ref4","doi-asserted-by":"publisher","DOI":"10.1145\/3131785.3131794"},{"key":"S0890060419000349_ref93","first-page":"3","volume-title":"Graphics Recognition Recent Advances. GREC 1999. Lecture Notes in Computer Science","volume":"1941","author":"Tombre","year":"2000"},{"key":"S0890060419000349_ref96","doi-asserted-by":"publisher","DOI":"10.1145\/3267782.3267788"},{"key":"S0890060419000349_ref14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2014.09.005"},{"key":"S0890060419000349_ref101","doi-asserted-by":"publisher","DOI":"10.3390\/rs10091461"},{"key":"S0890060419000349_ref102","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2019.8851809"},{"key":"S0890060419000349_ref103","doi-asserted-by":"publisher","DOI":"10.1145\/1873951.1873961"},{"key":"S0890060419000349_ref104","doi-asserted-by":"publisher","DOI":"10.1145\/1185657.1185770"},{"key":"S0890060419000349_ref105","unstructured":"Zhu Fan , Xie Jin and Fang Yi (2016) Learning cross-domain neural networks for sketch-based 3D shape retrieval. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16), Phoenix, Arizona, USA, February 12\u201317, 2016, pp. 3683\u20133689."},{"key":"S0890060419000349_ref40","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2009.05.005"},{"key":"S0890060419000349_ref31","unstructured":"Fiorentino M , De Amicis R , Stork A and Monno G (2002) Surface design in virtual reality as industrial application. Proceedings of the DESIGN Conference, May 14\u201317, 2002, Cavtat, Dubrovnik, Croatia, pp. 477\u2013482."},{"key":"S0890060419000349_ref64","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-35495-8_7"},{"key":"S0890060419000349_ref98","unstructured":"Wang F , Kang L and Li Y (2015) Sketch-based 3D shape retrieval using convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 7\u201312 June 2015, Boston, MA, USA."},{"key":"S0890060419000349_ref82","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s00163-002-0028-7","article-title":"Support value of sketching in the design process","volume":"14","author":"Sch\u00fctze","year":"2003","journal-title":"Research in Engineering Design"}],"container-title":["Artificial Intelligence for Engineering Design, Analysis and Manufacturing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0890060419000349","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T05:22:03Z","timestamp":1665120123000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0890060419000349\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":106,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,11]]}},"alternative-id":["S0890060419000349"],"URL":"https:\/\/doi.org\/10.1017\/s0890060419000349","relation":{},"ISSN":["0890-0604","1469-1760"],"issn-type":[{"value":"0890-0604","type":"print"},{"value":"1469-1760","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}