{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T21:44:38Z","timestamp":1757540678742,"version":"3.37.3"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2020,6,11]],"date-time":"2020-06-11T00:00:00Z","timestamp":1591833600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Brazilian National Council for the Improvement of Higher Education Personnel","award":["88\/2015"],"award-info":[{"award-number":["88\/2015"]}]},{"name":"National Institute of Science and Technology - Medicine Assisted by Scientific Computing","award":["2016\u20132021"],"award-info":[{"award-number":["2016\u20132021"]}]},{"DOI":"10.13039\/501100003593","name":"Conselho Nacional de Desenvolvimento Cient\u00edfico e Tecnol\u00f3gico","doi-asserted-by":"crossref","award":["308615\/2018-2"],"award-info":[{"award-number":["308615\/2018-2"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,5,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>As software systems have increased in complexity, manual testing has become harder or even infeasible. In addition, each test phase and application domain may have its idiosyncrasies in relation to testing automation. Techniques and tools to automate test oracles in domains such as graphical user interfaces are available; nevertheless, they are scarce in the virtual reality (VR) realm. We present an approach to automate software testing in VR-based systems with haptic interfaces\u2014interfaces that allow bidirectional communication during human\u2013computer interaction, capturing movements and providing touch feedback. It deals with the complexity and characteristics of haptic interfaces to apply the record and playback technique. Our approach also provides inference rules to identify possible faulty modules of the system under testing. A case study was performed with three systems: a system with primitive virtual objects, a dental anesthesia simulator and a game. Faulty versions of the systems were created by seeding faults manually and by using mutation operators. The results showed that 100% of the manually seeded faults and 93% of mutants were detected. Moreover, the inference rules helped identify the faulty modules of the systems, suggesting that the approach improves the test activity in VR-based systems with haptic interfaces.<\/jats:p>","DOI":"10.1093\/comjnl\/bxaa054","type":"journal-article","created":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T11:15:15Z","timestamp":1588245315000},"page":"826-841","source":"Crossref","is-referenced-by-count":6,"title":["Software Testing Automation of VR-Based Systems With Haptic Interfaces"],"prefix":"10.1093","volume":"64","author":[{"given":"Cl\u00e9ber G","family":"Corr\u00eaa","sequence":"first","affiliation":[{"name":"Computer Department, Universidade Tecnol\u00f3gica Federal do Paran\u00e1, 1640 Alberto Carazzai Avenue, Downtown, Corn\u00e9lio Proc\u00f3pio 86.300-000, Brazil"}]},{"given":"M\u00e1rcio E","family":"Delamaro","sequence":"additional","affiliation":[{"name":"Department of Computer Systems, Instituto de Ci\u00eancias Matem\u00e1ticas e de Computa\u00e7\u00e3o, Universidade de S\u00e3o Paulo, 400 Trabalhador S\u00e3o-carlense Avenue, Downtown, S\u00e3o Carlos 13.566-590, Brazil"}]},{"given":"Marcos L","family":"Chaim","sequence":"additional","affiliation":[{"name":"School of Arts, Sciences and Humanities, Universidade de S\u00e3o Paulo, 1000 Arlindo B\u00e9ttio Street, Vila Guaraciaba, S\u00e3o Paulo 03.828-000, Brazil"}]},{"given":"F\u00e1tima L S","family":"Nunes","sequence":"additional","affiliation":[{"name":"School of Arts, Sciences and Humanities, Universidade de S\u00e3o Paulo, 1000 Arlindo B\u00e9ttio Street, Vila Guaraciaba, S\u00e3o Paulo 03.828-000, Brazil"}]}],"member":"286","published-online":{"date-parts":[[2020,6,11]]},"reference":[{"volume-title":"3D User Interfaces: Theory and Practice","year":"2005","author":"Bowman","key":"2021052810194693300_ref1"},{"key":"2021052810194693300_ref2","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1145\/310930.310976","article-title":"Making graphics physically tangible","volume":"42","author":"Salisbury","year":"1999","journal-title":"Commun. ACM"},{"key":"2021052810194693300_ref3","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1590\/1678-7757-2016-0386","article-title":"Virtual reality simulator for dental anesthesia training in the inferior alveolar nerve block","volume":"25","author":"Corr\u00eaa","year":"2017","journal-title":"J. Appl. Oral Sci."},{"key":"2021052810194693300_ref4","first-page":"267","article-title":"Virtual Reality-Based System for Training in Dental Anesthesia","volume-title":"Proc. 16th Int. Conf. Human\u2013Computer Interaction (HCII)","author":"Corr\u00eaa","year":"2014"},{"key":"2021052810194693300_ref5","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1145\/2851613.2851688","article-title":"Haptic Interaction Objective Evaluation in Needle Insertion Task Simulation","volume-title":"Proc. ACM 31st Symposium on Applied Computing (SAC)","author":"Corr\u00eaa","year":"2016"},{"key":"2021052810194693300_ref6","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s10515-013-0128-9","article-title":"Guitar: An innovative tool for automated testing of GUI-driven software","volume":"21","author":"Nguyen","year":"2014","journal-title":"Autom. Softw. Eng."},{"key":"2021052810194693300_ref7","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1007\/s11219-011-9135-x","article-title":"Automated GUI performance testing","volume":"19","author":"Adamoli","year":"2011","journal-title":"Softw. Qual. J."},{"key":"2021052810194693300_ref8","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1016\/j.medengphy.2018.11.002","article-title":"Haptic interaction for needle insertion training in medical applications: The state-of-the-art","volume":"63","author":"Corr\u00eaa","year":"2019","journal-title":"Med. Eng. Phys."},{"key":"2021052810194693300_ref9","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1109\/TSE.2014.2372785","article-title":"The oracle problem in software testing: A survey","volume":"41","author":"Barr","year":"2015","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194693300_ref10","doi-asserted-by":"crossref","DOI":"10.1017\/9781316771273","volume-title":"Introduction to Software Testing","author":"Ammann","year":"2016"},{"key":"2021052810194693300_ref11","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/TSE.2016.2597136","article-title":"Test oracle strategies for model-based testing","volume":"43","author":"Li","year":"2017","journal-title":"IEEE Trans. Softw. Eng."},{"key":"2021052810194693300_ref12","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/TOH.2010.19","article-title":"The role of haptics in medical training simulators: A survey of the state of the art","volume":"4","author":"Coles","year":"2011","journal-title":"IEEE Trans. Haptics"},{"key":"2021052810194693300_ref13","first-page":"570","article-title":"QoE Assessment of Operability and Fairness for Soft Objects in Networked Real-Time Game With Haptic Sense","volume-title":"Proc. 21st Asia-Pacific Conf. Communications (APCC)","author":"Sithu","year":"2015"},{"key":"2021052810194693300_ref14","first-page":"1093","article-title":"Evaluating Playability on Haptic User Interface for Mobile Gaming","volume-title":"Proc. Int. Symposium on Information Technology","author":"Wong","year":"2010"},{"key":"2021052810194693300_ref15","doi-asserted-by":"crossref","DOI":"10.1162\/105474603322955950","volume-title":"Virtual Reality Technology","author":"Burdea","year":"2003"},{"key":"2021052810194693300_ref16","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1097\/MOU.0b013e32831a478c","article-title":"Haptic feedback in robot-assisted minimally invasive surgery","volume":"19","author":"Okamura","year":"2009","journal-title":"Curr. Opin. Urol."},{"key":"2021052810194693300_ref17","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1109\/TOH.2015.2445768","article-title":"Direct visuo-haptic 4D volume rendering using respiratory motion models","volume":"8","author":"Fortmeier","year":"2015","journal-title":"IEEE Transactions on Haptics"},{"key":"2021052810194693300_ref18","first-page":"7","article-title":"Haptic Simulation of Needle and Probe Interaction With Tissue for Prostate Brachytherapy Training","volume-title":"Proc. IEEE World Haptics Conf. (WHC)","author":"Goksel","year":"2011"},{"key":"2021052810194693300_ref19","first-page":"3459","article-title":"Towards an Augmented Ultrasound Guided Spinal Needle Insertion System","volume-title":"Annual Int. Conf. IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Sutherland","year":"2011"},{"key":"2021052810194693300_ref20","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1109\/TBME.2005.847548","article-title":"Interactive simulation of needle insertion models","volume":"52","author":"Dimaio","year":"2005","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"2021052810194693300_ref21","doi-asserted-by":"crossref","first-page":"1781","DOI":"10.1007\/s10916-010-9638-1","article-title":"Virtual suturing simulation based on commodity physics engine for medical learning","volume":"36","author":"Choi","year":"2012","journal-title":"J. Med. Syst."},{"article-title":"Introduction to jfcUnit","year":"2004","author":"Caswell","key":"2021052810194693300_ref22"},{"key":"2021052810194693300_ref23","first-page":"1","article-title":"Test Oracles Based on Metamorphic Relations for Image Processing Applications","volume-title":"Proc. 16th IEEE\/ACIS Int. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing (SNPD)","author":"Jameel","year":"2015"},{"key":"2021052810194693300_ref24","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1002\/stvr.463","article-title":"Using concepts of content-based image retrieval to implement graphical testing oracles","volume":"23","author":"Delamaro","year":"2013","journal-title":"Softw. Test. Verif. Reliab."},{"key":"2021052810194693300_ref25","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/s10664-012-9216-x","article-title":"A case study on bypass testing of web applications","volume":"19","author":"Offutt","year":"2014","journal-title":"Empirical Engineering"},{"key":"2021052810194693300_ref26","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1002\/stvr.437","article-title":"Automated functional testing of online search services","volume":"22","author":"Zhou","year":"2010","journal-title":"Softw. Test. Verif. Reliab."},{"key":"2021052810194693300_ref27","first-page":"180","article-title":"Why Do Record\/Replay Tests of Web Applications Break?","volume-title":"Proc. IEEE Int. Conf. Software Testing, Verification and Validation (ICST)","author":"Hammoudi","year":"2016"},{"key":"2021052810194693300_ref28","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1145\/1101908.1101947","article-title":"Automated Replay and Failure Detection for Web Applications","volume-title":"Proc. 20th IEEE\/ACM Int. Conf. Automated Software Engineering","author":"Sprenkle","year":"2005"},{"key":"2021052810194693300_ref29","first-page":"56","article-title":"Definition of Test Criteria Based on the Scene Graph for VR Applications","volume-title":"Proc. XIII Symposium on Virtual Reality","author":"Bezerra","year":"2011"},{"key":"2021052810194693300_ref30","first-page":"1","article-title":"An automated functional testing approach for virtual reality applications","volume":"28","author":"Souza","year":"2018","journal-title":"Softw. Test. Verif. Reliab."},{"volume-title":"Kina: An enhanced development model and toolkit for kinect applications","year":"2013","author":"de Souza","key":"2021052810194693300_ref31"},{"key":"2021052810194693300_ref32","first-page":"107","article-title":"Automated Testing of Virtual Reality Application Interfaces","volume-title":"Proc. Workshop on Virtual Environments (EGVE)","author":"Bierbaum","year":"2003"},{"key":"2021052810194693300_ref33","first-page":"123","article-title":"Automatic Testing of Natural User Interfaces","volume-title":"Proc. IEEE Seventh Int. Conf. Software Testing, Verification and Validation","author":"Hunt","year":"2014"},{"key":"2021052810194693300_ref34","first-page":"204","article-title":"Modelling Hand Gestures to Test Leap Motion Controlled Applications","volume-title":"Proc. IEEE Int. Conf. Software Testing, Verification and Validation Workshops (ICSTW)","author":"White","year":"2018"},{"volume-title":"Software Test Automation: Effective Use of Test Execution Tools","year":"1999","author":"Fewster","key":"2021052810194693300_ref35"},{"key":"2021052810194693300_ref36","first-page":"63","article-title":"Haptic Simulation for Virtual Training in Application of Dental Anesthesia","volume-title":"Proc. XV Symposium on Virtual and Augmented Reality (SVR)","author":"Corr\u00eaa","year":"2013"},{"key":"2021052810194693300_ref37","first-page":"12","article-title":"ViMeTGame: A serious game for virtual medical training of breast biopsy","volume":"3","author":"Torres","year":"2013","journal-title":"SBC J. 3D Interact. Syst."},{"article-title":"Touch","year":"2017","author":"3DSystems","key":"2021052810194693300_ref38"},{"volume-title":"Aspect-Oriented Programming with AspectJ","year":"2002","author":"Kiselev","key":"2021052810194693300_ref39"},{"volume-title":"Computer Graphics, C Version","year":"1997","author":"Hearn","key":"2021052810194693300_ref40"},{"key":"2021052810194693300_ref41","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/2945.675649","article-title":"Efficient collision detection using bounding volume hierarchies of k-DOPs","volume":"4","author":"Klosowski","year":"1998","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"article-title":"Mujava home page","year":"2013","author":"Offut","key":"2021052810194693300_ref42"},{"key":"2021052810194693300_ref43","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1002\/stvr.308","article-title":"Mujava: An automated class mutation system","volume":"15","author":"Ma","year":"2005","journal-title":"Softw. Test. Verif. Reliab."},{"article-title":"Mutation Analysis as a Tool for Software Quality Assurance","year":"1980","author":"DeMillo","key":"2021052810194693300_ref44"}],"container-title":["The Computer Journal"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/5\/826\/38335120\/bxaa054.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/5\/826\/38335120\/bxaa054.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,28]],"date-time":"2021-05-28T11:55:27Z","timestamp":1622202927000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/comjnl\/article\/64\/5\/826\/5856158"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,11]]},"references-count":44,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,6,11]]},"published-print":{"date-parts":[[2021,5,19]]}},"URL":"https:\/\/doi.org\/10.1093\/comjnl\/bxaa054","relation":{},"ISSN":["0010-4620","1460-2067"],"issn-type":[{"type":"print","value":"0010-4620"},{"type":"electronic","value":"1460-2067"}],"subject":[],"published-other":{"date-parts":[[2021,5]]},"published":{"date-parts":[[2020,6,11]]}}}