{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T17:48:45Z","timestamp":1772905725659,"version":"3.50.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,7,19]],"date-time":"2021-07-19T00:00:00Z","timestamp":1626652800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Samsung Research Funding Center of Samsung Electronics","award":["SRFC-IT2001-04"],"award-info":[{"award-number":["SRFC-IT2001-04"]}]},{"name":"Korea NRF grant","award":["2019R1A2C3007229"],"award-info":[{"award-number":["2019R1A2C3007229"]}]},{"name":"MSRA"},{"name":"MSIT\/IITP of Korea","award":["2017-0-00072"],"award-info":[{"award-number":["2017-0-00072"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,8,31]]},"abstract":"<jats:p>Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. The size of a message or identification code is limited by the spatial resolution of binary patterns in a marker. Markers often fail to be recognized due to various imaging artifacts of optical\/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.<\/jats:p>","DOI":"10.1145\/3450626.3459762","type":"journal-article","created":{"date-parts":[[2021,7,20]],"date-time":"2021-07-20T00:04:26Z","timestamp":1626739466000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["DeepFormableTag"],"prefix":"10.1145","volume":"40","author":[{"given":"Mustafa B.","family":"Yaldiz","sequence":"first","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Meuleman","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hyeonjoong","family":"Jang","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hyunho","family":"Ha","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Min H.","family":"Kim","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,7,19]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"The Conference and Workshop on Neural Information Processing Systems. 2069--2079","author":"Baluja Shumeet","year":"2017","unstructured":"Shumeet Baluja . 2017 . Hiding images in plain sight: Deep steganography . In The Conference and Workshop on Neural Information Processing Systems. 2069--2079 . Shumeet Baluja. 2017. Hiding images in plain sight: Deep steganography. In The Conference and Workshop on Neural Information Processing Systems. 2069--2079."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.475"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995544"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.164"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_2_6_1","unstructured":"Denso Wave. 1994. Quick Response (QR) code. https:\/\/d1wqtxts1xzle7.cloudfront.net\/51791265\/Three_QR_Code.pdf  Denso Wave. 1994. Quick Response (QR) code. https:\/\/d1wqtxts1xzle7.cloudfront.net\/51791265\/Three_QR_Code.pdf"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2018.00060"},{"key":"e_1_2_2_8_1","volume-title":"Constructive Theory of Functions of Several Variables","author":"Duchon Jean","unstructured":"Jean Duchon . 1977. Splines minimizing rotation-invariant semi-norms in Sobolev spaces . In Constructive Theory of Functions of Several Variables , Walter Schempp and Karl Zeller (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 85--100. Jean Duchon. 1977. Splines minimizing rotation-invariant semi-norms in Sobolev spaces. In Constructive Theory of Functions of Several Variables, Walter Schempp and Karl Zeller (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 85--100."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.74"},{"key":"e_1_2_2_10_1","volume-title":"Lens distortion for close-range photogrammetry. Photogrammetric engineering and remote sensing 52, 1","author":"Fryer John G","year":"1986","unstructured":"John G Fryer and Duane C Brown . 1986. Lens distortion for close-range photogrammetry. Photogrammetric engineering and remote sensing 52, 1 ( 1986 ), 51--58. John G Fryer and Duane C Brown. 1986. Lens distortion for close-range photogrammetry. Photogrammetric engineering and remote sensing 52, 1 (1986), 51--58."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.01.005"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2015.09.023"},{"key":"e_1_2_2_13_1","volume-title":"The Conference and Workshop on Neural Information Processing Systems. 4143--4151","author":"Grinchuk Oleg","year":"2016","unstructured":"Oleg Grinchuk , Vadim Lebedev , and Victor Lempitsky . 2016 . Learnable visual markers . In The Conference and Workshop on Neural Information Processing Systems. 4143--4151 . Oleg Grinchuk, Vadim Lebedev, and Victor Lempitsky. 2016. Learnable visual markers. In The Conference and Workshop on Neural Information Processing Systems. 4143--4151."},{"key":"e_1_2_2_14_1","volume-title":"The Conference and Workshop on Neural Information Processing Systems. 1954--1963","author":"Hayes Jamie","year":"2017","unstructured":"Jamie Hayes and George Danezis . 2017 . Generating steganographic images via adversarial training . In The Conference and Workshop on Neural Information Processing Systems. 1954--1963 . Jamie Hayes and George Danezis. 2017. Generating steganographic images via adversarial training. In The Conference and Workshop on Neural Information Processing Systems. 1954--1963."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.322"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00863"},{"key":"e_1_2_2_18_1","volume-title":"Spatial Transformer Networks. In The Conference and Workshop on Neural Information Processing Systems. 2017--2025","author":"Jaderberg Max","year":"2015","unstructured":"Max Jaderberg , Karen Simonyan , Andrew Zisserman , and Koray Kavukcuoglu . 2015 . Spatial Transformer Networks. In The Conference and Workshop on Neural Information Processing Systems. 2017--2025 . http:\/\/papers.nips.cc\/paper\/5854-spatial-transformer-networks Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. 2015. Spatial Transformer Networks. In The Conference and Workshop on Neural Information Processing Systems. 2017--2025. http:\/\/papers.nips.cc\/paper\/5854-spatial-transformer-networks"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1435731.1437872"},{"key":"e_1_2_2_20_1","volume-title":"Determining and Improving the Localization Accuracy of AprilTag Detection. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 8288--8294","author":"Kallwies Jan","year":"2020","unstructured":"Jan Kallwies , Bianca Forkel , and Hans-Joachim Wuensche . 2020 . Determining and Improving the Localization Accuracy of AprilTag Detection. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 8288--8294 . Jan Kallwies, Bianca Forkel, and Hans-Joachim Wuensche. 2020. Determining and Improving the Localization Accuracy of AprilTag Detection. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 8288--8294."},{"key":"e_1_2_2_21_1","volume-title":"International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Hk99zCeAb","author":"Karras Tero","year":"2018","unstructured":"Tero Karras , Timo Aila , Samuli Laine , and Jaakko Lehtinen . 2018 . Progressive Growing of GANs for Improved Quality, Stability, and Variation . In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Hk99zCeAb Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Hk99zCeAb"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00453"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWAR.1999.803809"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967787"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2019.00103"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.106"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01252-6_6"},{"key":"e_1_2_2_29_1","volume-title":"Aruco: a minimal library for augmented reality applications based on opencv. Universidad de C\u00f3rdoba","author":"Munoz-Salinas Rafael","year":"2012","unstructured":"Rafael Munoz-Salinas . 2012. Aruco: a minimal library for augmented reality applications based on opencv. Universidad de C\u00f3rdoba ( 2012 ). Rafael Munoz-Salinas. 2012. Aruco: a minimal library for augmented reality applications based on opencv. Universidad de C\u00f3rdoba (2012)."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2002.1115065"},{"key":"e_1_2_2_31_1","volume-title":"Dynamic projection mapping onto deforming non-rigid surface using deformable dot cluster marker","author":"Narita Gaku","year":"2016","unstructured":"Gaku Narita , Yoshihiro Watanabe , and Masatoshi Ishikawa . 2016. Dynamic projection mapping onto deforming non-rigid surface using deformable dot cluster marker . IEEE transactions on visualization and computer graphics 23, 3 ( 2016 ), 1235--1248. Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2016. Dynamic projection mapping onto deforming non-rigid surface using deformable dot cluster marker. IEEE transactions on visualization and computer graphics 23, 3 (2016), 1235--1248."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5979561"},{"key":"e_1_2_2_33_1","unstructured":"OpenCV. 2020. Open Source Computer Vision Library. https:\/\/opencv.org\/. Version 4.2.0.  OpenCV. 2020. Open Source Computer Vision Library. https:\/\/opencv.org\/. Version 4.2.0."},{"key":"e_1_2_2_34_1","volume-title":"British Machine Vision Conference (BMVC).","author":"Peace John","unstructured":"John Peace , Eric Psota , Yanfeng Liu , and Lance C. P\u00e9rez . 2020. E2ETag: An End-to-End Trainable Method for Generating and Detecting Fiducial Markers . In British Machine Vision Conference (BMVC). John Peace, Eric Psota, Yanfeng Liu, and Lance C. P\u00e9rez. 2020. E2ETag: An End-to-End Trainable Method for Generating and Detecting Fiducial Markers. In British Machine Vision Conference (BMVC)."},{"key":"e_1_2_2_35_1","volume-title":"The Conference and Workshop on Neural Information Processing Systems. 91--99","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . 2015 . Faster r-cnn: Towards real-time object detection with region proposal networks . In The Conference and Workshop on Neural Information Processing Systems. 91--99 . Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In The Conference and Workshop on Neural Information Processing Systems. 91--99."},{"key":"e_1_2_2_36_1","volume-title":"Speeded up detection of squared fiducial markers. Image and vision Computing 76","author":"Romero-Ramirez Francisco J","year":"2018","unstructured":"Francisco J Romero-Ramirez , Rafael Mu\u00f1oz-Salinas , and Rafael Medina-Carnicer . 2018. Speeded up detection of squared fiducial markers. Image and vision Computing 76 ( 2018 ), 38--47. Francisco J Romero-Ramirez, Rafael Mu\u00f1oz-Salinas, and Rafael Medina-Carnicer. 2018. Speeded up detection of squared fiducial markers. Image and vision Computing 76 (2018), 38--47."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00219"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2017.2745572"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2011.6092394"},{"key":"e_1_2_2_40_1","volume-title":"Microfacet Models for Refraction through Rough Surfaces. Rendering techniques 2007","author":"Walter Bruce","year":"2007","unstructured":"Bruce Walter , Stephen R Marschner , Hongsong Li , and Kenneth E Torrance . 2007. Microfacet Models for Refraction through Rough Surfaces. Rendering techniques 2007 ( 2007 ), 18th. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. Rendering techniques 2007 (2007), 18th."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759617"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00161"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.3390\/fi10060054"},{"key":"e_1_2_2_44_1","unstructured":"Yuxin Wu Alexander Kirillov Francisco Massa Wan-Yen Lo and Ross Girshick. 2019. Detectron2. https:\/\/github.com\/facebookresearch\/detectron2.  Yuxin Wu Alexander Kirillov Francisco Massa Wan-Yen Lo and Ross Girshick. 2019. Detectron2. https:\/\/github.com\/facebookresearch\/detectron2."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CRV.2011.13"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00577"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01267-0_40"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3450626.3459762","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3450626.3459762","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:16Z","timestamp":1750191436000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3450626.3459762"}},"subtitle":["end-to-end generation and recognition of deformable fiducial markers"],"short-title":[],"issued":{"date-parts":[[2021,7,19]]},"references-count":47,"aliases":["10.1145\/3476576.3476619"],"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,8,31]]}},"alternative-id":["10.1145\/3450626.3459762"],"URL":"https:\/\/doi.org\/10.1145\/3450626.3459762","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,19]]},"assertion":[{"value":"2021-07-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}