{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T12:57:17Z","timestamp":1753880237787,"version":"3.41.2"},"reference-count":103,"publisher":"ASME International","issue":"12","license":[{"start":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T00:00:00Z","timestamp":1725840000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.asme.org\/publications-submissions\/publishing-information\/legal-policies"}],"content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Computer vision (CV) algorithms require large annotated datasets that are often labor-intensive and expensive to create. We propose AnnotateXR, an extended reality (XR) workflow to collect various high-fidelity data and auto-annotate it in a single demonstration. AnnotateXR allows users to align virtual models over physical objects, tracked with six degrees-of-freedom (6DOF) sensors. AnnotateXR utilizes a hand tracking capable XR head-mounted display coupled with 6DOF information and collision detection to enable algorithmic segmentation of different actions in videos through its digital twin. The virtual\u2013physical mapping provides a tight bounding volume to generate semantic segmentation masks for the captured image data. Alongside supporting object and action segmentation, we also support other dimensions of annotation required by modern CV, such as human\u2013object, object\u2013object, and rich 3D recordings, all with a single demonstration. Our user study shows AnnotateXR produced over 112,000 annotated data points in 67 min.<\/jats:p>","DOI":"10.1115\/1.4066180","type":"journal-article","created":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T13:46:04Z","timestamp":1723038364000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":1,"title":["AnnotateXR: An Extended Reality Workflow for Automating Data Annotation to Support Computer Vision Applications"],"prefix":"10.1115","volume":"24","author":[{"given":"Subramanian","family":"Chidambaram","sequence":"first","affiliation":[{"name":"Purdue University School of Mechanical Engineering, , West Lafayette, IN 47907"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rahul","family":"Jain","sequence":"additional","affiliation":[{"name":"Purdue University School of Electrical and Computer Engineering, , West Lafayette, IN 47907"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sai","family":"Swarup Reddy","sequence":"additional","affiliation":[{"name":"Purdue University School of Mechanical Engineering, , West Lafayette, IN 47907"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Asim","family":"Unmesh","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/02dqehb95","id-type":"ROR","asserted-by":"publisher"}],"name":"Purdue University West Lafayette School of Electrical and Computer Engineering, , West Lafayette, IN 47907"},{"name":"Purdue University School of Electrical and Computer Engineering, , West Lafayette, IN 47907"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karthik","family":"Ramani","sequence":"additional","affiliation":[{"name":"Purdue University School of Mechanical Engineering, , West Lafayette, IN 47907"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"33","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"2024090917335818000_CIT0001","first-page":"954","article-title":"The Apolloscape Dataset for Autonomous Driving","author":"Huang","year":"2018"},{"key":"2024090917335818000_CIT0002","first-page":"163","article-title":"Towards Fully Autonomous Driving: Systems and Algorithms","author":"Levinson","year":"2011"},{"key":"2024090917335818000_CIT0003","first-page":"234","article-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation","author":"Ronneberger","year":"2015"},{"issue":"11","key":"2024090917335818000_CIT0004","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision Meets Robotics: The Kitti Dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Rob. Res."},{"issue":"3","key":"2024090917335818000_CIT0005","doi-asserted-by":"publisher","first-page":"031005","DOI":"10.1115\/1.4062970","article-title":"The Design of a Virtual Prototyping System for Authoring Interactive Virtual Reality Environments From Real-World Scans","volume":"24","author":"Ipsita","year":"2023","journal-title":"ASME J. Comput. Inf. Sci. Eng."},{"key":"2024090917335818000_CIT0006","doi-asserted-by":"crossref","DOI":"10.1145\/3411763.3451747","article-title":"VRFromX: From Scanned Reality to Interactive Virtual Experience With Human-In-The-Loop","author":"Ipsita","year":"2021"},{"key":"2024090917335818000_CIT0007","first-page":"248","article-title":"ImageNet: A Large-Scale Hierarchical Image Database","author":"Deng","year":"2009"},{"year":"2016","author":"Abu-El-Haija","key":"2024090917335818000_CIT0008"},{"year":"2015","author":"Chang","key":"2024090917335818000_CIT0009"},{"key":"2024090917335818000_CIT0010","first-page":"2556","article-title":"HMDB: A Large Video Database for Human Motion Recognition","author":"Kuehne","year":"2011"},{"key":"2024090917335818000_CIT0011","first-page":"11184","article-title":"Home Action Genome: Cooperative Compositional Action Understanding","author":"Rai","year":"2021"},{"year":"2005","author":"Amazon","key":"2024090917335818000_CIT0012"},{"year":"2018","author":"Amazon Web Services","key":"2024090917335818000_CIT0013"},{"year":"2017","author":"Supervisely","key":"2024090917335818000_CIT0014"},{"year":"2016","author":"Anolytics","key":"2024090917335818000_CIT0015"},{"key":"2024090917335818000_CIT0016","first-page":"237","article-title":"Crowdsourcing High Quality Labels With a Tight Budget","author":"Li","year":"2016"},{"issue":"4","key":"2024090917335818000_CIT0017","doi-asserted-by":"publisher","first-page":"1448","DOI":"10.1109\/TNNLS.2020.2984729","article-title":"Active Multilabel Crowd Consensus","volume":"32","author":"Yu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"2024090917335818000_CIT0018","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.01025","article-title":"Action Genome: Actions as Compositions of Spatio-Temporal Scene Graphs","author":"Ji","year":"2020"},{"key":"2024090917335818000_CIT0019","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1007\/978-3-319-46475-6_7","volume-title":"Computer Vision \u2013 ECCV 2016","author":"Richter","year":"2016"},{"key":"2024090917335818000_CIT0020","first-page":"1666","article-title":"O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning","author":"Mo","year":"2022"},{"article-title":"Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects","year":"2018","author":"Tremblay","key":"2024090917335818000_CIT0021"},{"issue":"2","key":"2024090917335818000_CIT0022","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1016\/j.tics.2021.11.008","article-title":"Next-Generation Deep Learning Based on Simulators and Synthetic Data","volume":"26","author":"de Melo","year":"2022","journal-title":"Trends Cogn. Sci."},{"issue":"3","key":"2024090917335818000_CIT0023","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1016\/j.vrih.2021.05.002","article-title":"Survey on Depth and RGB Image-Based 3D Hand Shape and Pose Estimation","volume":"3","author":"Huang","year":"2021","journal-title":"Virtual Real. Intell. Hardw."},{"key":"2024090917335818000_CIT0024","first-page":"779","article-title":"You Only Look Once: Unified, Real-Time Object Detection","author":"Redmon","year":"2016"},{"key":"2024090917335818000_CIT0025","first-page":"3431","article-title":"Fully Convolutional Networks for Semantic Segmentation","author":"Long","year":"2015"},{"key":"2024090917335818000_CIT0026","first-page":"1049","article-title":"Temporal Action Localization in Untrimmed Videos Via Multi-stage CNNS","author":"Shou","year":"2016"},{"key":"2024090917335818000_CIT0027","first-page":"4561","article-title":"Pvnet: Pixel-Wise Voting Network for 6DOF Pose Estimation","author":"Peng","year":"2019"},{"key":"2024090917335818000_CIT0028","first-page":"10138","article-title":"H2O: Two Hands Manipulating Objects for First Person Interaction Recognition","author":"Kwon","year":"2021"},{"issue":"1","key":"2024090917335818000_CIT0029","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1109\/LRA.2023.3332554","article-title":"Interacting Objects: A Dataset of Object-Object Interactions for Richer Dynamic Scene Representations","volume":"9","author":"Unmesh","year":"2024","journal-title":"IEEE Rob. Autom. Lett."},{"key":"2024090917335818000_CIT0030","first-page":"414","article-title":"Atlas: End-to-End 3D Scene Reconstruction From Posed Images","author":"Murez","year":"2020"},{"issue":"1","key":"2024090917335818000_CIT0031","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40323-020-00147-4","article-title":"How to Tell the Difference Between a Model and a Digital Twin","volume":"7","author":"Wright","year":"2020","journal-title":"Adv. Model. Simul. Eng. Sci."},{"year":"2018","author":"Hughes","key":"2024090917335818000_CIT0032"},{"year":"2021","author":"Antilatency","key":"2024090917335818000_CIT0033"},{"year":"2020","author":"Oculus","key":"2024090917335818000_CIT0034"},{"key":"2024090917335818000_CIT0035","doi-asserted-by":"crossref","DOI":"10.1145\/3586183.3606793","article-title":"Ubi-TOUCH: Ubiquitous Tangible Object Utilization Through Consistent Hand-Object Interaction in Augmented Reality","author":"Jain","year":"2023"},{"year":"2014","author":"Scheff-King","key":"2024090917335818000_CIT0036"},{"year":"2022","author":"Stratasys","key":"2024090917335818000_CIT0037"},{"year":"1990","author":"Traceparts","key":"2024090917335818000_CIT0038"},{"year":"2021","author":"CVML","key":"2024090917335818000_CIT0039"},{"year":"2018","author":"Yu","key":"2024090917335818000_CIT0040"},{"year":"2017","author":"Kawaguchi","key":"2024090917335818000_CIT0041"},{"key":"2024090917335818000_CIT0042","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.690","article-title":"YOLO9000: Better, Faster, Stronger","author":"Redmon","year":"2017"},{"year":"2018","author":"Redmon","key":"2024090917335818000_CIT0043"},{"year":"2021","author":"Liu","key":"2024090917335818000_CIT0044"},{"key":"2024090917335818000_CIT0045","first-page":"740","article-title":"Microsoft COCO: Common Objects in Context","author":"Lin","year":"2014"},{"year":"2019","author":"Microsoft","key":"2024090917335818000_CIT0046"},{"year":"2020","author":"Microsoft","key":"2024090917335818000_CIT0047"},{"year":"2020","author":"Microsoft","key":"2024090917335818000_CIT0048"},{"year":"2015","author":"Microsoft","key":"2024090917335818000_CIT0049"},{"year":"2020","author":"Microsoft","key":"2024090917335818000_CIT0050"},{"year":"2017","author":"Anolytics","key":"2024090917335818000_CIT0051"},{"key":"2024090917335818000_CIT0052","first-page":"549","article-title":"What\u2019s The Point: Semantic Segmentation With Point Supervision","author":"Bearman","year":"2016"},{"key":"2024090917335818000_CIT0053","first-page":"987","article-title":"Labelar: A Spatial Guidance Interface for Fast Computer Vision Image Collection","author":"Laielli","year":"2019"},{"issue":"2","key":"2024090917335818000_CIT0054","doi-asserted-by":"publisher","first-page":"1179","DOI":"10.1109\/LRA.2016.2532924","article-title":"A Dataset for Improved RGBD-Based Object Detection and Pose Estimation for Warehouse Pick-and-Place","volume":"1","author":"Rennie","year":"2016","journal-title":"IEEE Rob. Autom. Lett."},{"key":"2024090917335818000_CIT0055","first-page":"582","article-title":"A Framework for Evaluating 6-DOF Object Trackers","author":"Garon","year":"2018"},{"key":"2024090917335818000_CIT0056","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1007\/978-3-030-58548-8_34","article-title":"GRAB: A Dataset of Whole-Body Human Grasping of Objects","volume-title":"Computer Vision \u2013 ECCV 2020","author":"Taheri","year":"2020"},{"key":"2024090917335818000_CIT0057","first-page":"601","article-title":"Recovering Accurate 3D Human Pose in the Wild Using IMUS and a Moving Camera","author":"von Marcard","year":"2018"},{"key":"2024090917335818000_CIT0058","first-page":"3198","article-title":"Beat the Mturkers: Automatic Image Labeling From Weak 3D Supervision","author":"Chen","year":"2014"},{"key":"2024090917335818000_CIT0059","first-page":"3688","article-title":"Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer","author":"Xie","year":"2016"},{"key":"2024090917335818000_CIT0060","first-page":"5230","article-title":"Annotating Object Instances With a Polygon-RNN","author":"Castrejon","year":"2017"},{"key":"2024090917335818000_CIT0061","first-page":"859","article-title":"Efficient Interactive Annotation of Segmentation Datasets With Polygon-RNN++","author":"Acuna","year":"2018"},{"key":"2024090917335818000_CIT0062","first-page":"7822","article-title":"Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations","author":"Ahmadyan","year":"2021"},{"key":"2024090917335818000_CIT0063","doi-asserted-by":"crossref","DOI":"10.1145\/3526113.3545663","article-title":"Arnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation","author":"Qian","year":"2022"},{"key":"2024090917335818000_CIT0064","doi-asserted-by":"crossref","DOI":"10.1145\/3532719.3543249","article-title":"Immersive-Labeler: Immersive Annotation of Large-Scale 3D Point Clouds in Virtual Reality","author":"Doula","year":"2022"},{"key":"2024090917335818000_CIT0065","first-page":"1","article-title":"Gesture-Aware Interactive Machine Teaching With In-Situ Object Annotations","author":"Zhou","year":"2022"},{"article-title":"Scaling Egocentric Vision: The Epic-Kitchens Dataset","year":"2018","author":"Damen","key":"2024090917335818000_CIT0066"},{"year":"2018","author":"Sigurdsson","key":"2024090917335818000_CIT0067"},{"key":"2024090917335818000_CIT0068","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2015.7298698","article-title":"Activitynet: A Large-Scale Video Benchmark for Human Activity Understanding","author":"Caba Heilbron","year":"2015"},{"key":"2024090917335818000_CIT0069","first-page":"2408","article-title":"AVA: A Large-Scale Database for Aesthetic Visual Analysis","author":"Murray","year":"2012"},{"key":"2024090917335818000_CIT0070","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2019.00130","article-title":"Coin: A Large-Scale Dataset for Comprehensive Instructional Video Analysis","author":"Tang","year":"2019"},{"year":"2017","author":"Kay","key":"2024090917335818000_CIT0071"},{"key":"2024090917335818000_CIT0072","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.00113","article-title":"Something-Else: Compositional Action Recognition With Spatial-Temporal Interaction Networks","author":"Materzynska","year":"2020"},{"key":"2024090917335818000_CIT0073","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.00269","article-title":"Finegym: A Hierarchical Video Dataset for Fine-Grained Action Understanding","author":"Shao","year":"2020"},{"key":"2024090917335818000_CIT0074","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2016.85","article-title":"A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation","author":"Perazzi","year":"2016"},{"issue":"2","key":"2024090917335818000_CIT0075","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1016\/j.patrec.2008.04.005","article-title":"Semantic Object Classes in Video: A High-Definition Ground Truth Database","volume":"30","author":"Brostow","year":"2009","journal-title":"Pattern Recogn. Lett."},{"key":"2024090917335818000_CIT0076","first-page":"496","article-title":"Active Frame Selection for Label Propagation in Videos","author":"Vijayanarasimhan","year":"2012"},{"key":"2024090917335818000_CIT0077","first-page":"847","article-title":"The IKEA ASM Dataset: Understanding People Assembling Furniture Through Actions, Objects and Pose","author":"Ben-Shabat","year":"2021"},{"key":"2024090917335818000_CIT0078","first-page":"715","article-title":"Automatic Dataset Generation From CAD for Vision-Based Grasping","author":"Ahmad","year":"2021"},{"year":"2022","author":"Microsoft","key":"2024090917335818000_CIT0079"},{"year":"2022","author":"Inc., N.","key":"2024090917335818000_CIT0080"},{"year":"2021","author":"Stereolabs","key":"2024090917335818000_CIT0081"},{"key":"2024090917335818000_CIT0082","first-page":"234","article-title":"ProcessAR: An Augmented Reality-Based Tool to Create In-Situ Procedural 2D\/3D AR Instructions","author":"Chidambaram","year":"2021"},{"year":"2022","author":"Ramani","key":"2024090917335818000_CIT0083"},{"key":"2024090917335818000_CIT0084","first-page":"326","article-title":"Editar: A Digital Twin Authoring Environment for Creation of AR\/VR and Video Instructions From a Single Demonstration","author":"Chidambaram","year":"2022"},{"year":"2024","author":"Ramani","key":"2024090917335818000_CIT0085"},{"year":"2020","author":"qlone","key":"2024090917335818000_CIT0086"},{"year":"2020","author":"cognex","key":"2024090917335818000_CIT0087"},{"year":"2020","author":"display.land","key":"2024090917335818000_CIT0088"},{"year":"2020","author":"Unity","key":"2024090917335818000_CIT0089"},{"key":"2024090917335818000_CIT0090","first-page":"145","article-title":"Authoring Sensor-Based Interactions by Demonstration With Direct Manipulation and Pattern Recognition","author":"Hartmann","year":"2007"},{"issue":"194","key":"2024090917335818000_CIT0091","first-page":"4","article-title":"SUS-a Quick and Dirty Usability Scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Ind."},{"issue":"6","key":"2024090917335818000_CIT0092","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks","volume":"39","author":"Ren","year":"2016","journal-title":"IEEE Trans. Patt. Anal. Mach. Intell."},{"key":"2024090917335818000_CIT0093","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","volume-title":"Computer Vision \u2013 ECCV 2014","author":"Lin","year":"2014"},{"key":"2024090917335818000_CIT0094","first-page":"2961","article-title":"Mask R-CNN","author":"He","year":"2017"},{"key":"2024090917335818000_CIT0095","first-page":"36","article-title":"Segmental Spatiotemporal CNNS for Fine-Grained Action Segmentation","author":"Lea","year":"2016"},{"key":"2024090917335818000_CIT0096","first-page":"1961","article-title":"A Multi-Stream Bi-Directional Recurrent Neural Network for Fine-Grained Action Detection","author":"Singh","year":"2016"},{"key":"2024090917335818000_CIT0097","first-page":"780","article-title":"The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities","author":"Kuehne","year":"2014"},{"issue":"3","key":"2024090917335818000_CIT0098","first-page":"114","article-title":"Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale","volume":"4","author":"Bangor","year":"2009","journal-title":"J. Usabil. Stud."},{"key":"2024090917335818000_CIT0099","first-page":"1233","article-title":"Snaptoreality: Aligning Augmented Reality to the Real World","author":"Nuernberger","year":"2016"},{"key":"2024090917335818000_CIT0100","first-page":"1185","article-title":"Plane, Ray, and Point: Enabling Precise Spatial Manipulations With Shape Constraints","author":"Hayatpur","year":"2019"},{"key":"2024090917335818000_CIT0101","first-page":"1","article-title":"An Exploratory Study of Augmented Reality Presence for Tutoring Machine Tasks","author":"Cao","year":"2020"},{"key":"2024090917335818000_CIT0102","first-page":"665","article-title":"iSOFT: A Customizable Soft Sensor With Real-Time Continuous Contact and Stretching Sensing","author":"Yoon","year":"2017"},{"issue":"2","key":"2024090917335818000_CIT0103","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3463518","article-title":"FabHandWear: An End-to-End Pipeline From Design to Fabrication of Customized Functional Hand Wearables","volume":"5","author":"Paredes","year":"2021","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."}],"container-title":["Journal of Computing and Information Science in Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/24\/12\/121001\/7375799\/jcise_24_12_121001.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/24\/12\/121001\/7375799\/jcise_24_12_121001.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T17:34:18Z","timestamp":1725903258000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article\/24\/12\/121001\/1202055\/AnnotateXR-An-Extended-Reality-Workflow-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,9]]},"references-count":103,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,12,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.4066180","relation":{},"ISSN":["1530-9827","1944-7078"],"issn-type":[{"type":"print","value":"1530-9827"},{"type":"electronic","value":"1944-7078"}],"subject":[],"published":{"date-parts":[[2024,9,9]]},"article-number":"121001"}}