{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,10]],"date-time":"2025-12-10T08:54:07Z","timestamp":1765356847069,"version":"build-2065373602"},"reference-count":67,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2020,12,25]],"date-time":"2020-12-25T00:00:00Z","timestamp":1608854400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004955","name":"\u00d6sterreichische Forschungsf\u00f6rderungsgesellschaft","doi-asserted-by":"publisher","award":["871459"],"award-info":[{"award-number":["871459"]}],"id":[{"id":"10.13039\/501100004955","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>A novel approach to generic (or generalized) robot programming and a novel simplified, block-based programming environment, called \u201cAssembly\u201d, are introduced. The approach leverages the newest graphical user interface automation tools and techniques to generate programs in various proprietary robot programming environments by emulating user interactions in those environments. The \u201cAssembly\u201d tool is used to generate robot-independent intermediary program models, which are translated into robot-specific programs using a graphical user interface automation toolchain. The generalizability of the approach to list, tree, and block-based programming is assessed using three different robot programming environments, two of which are proprietary. The results of this evaluation suggest that the proposed approach is feasible for an entire range of programming models and thus enables the generation of programs in various proprietary robot programming environments. In educational settings, the automated generation of programs fosters learning different robot programming models by example. For experts, the proposed approach provides a means for generating program (or task) templates, which can be adjusted to the needs of the application at hand on the shop floor.<\/jats:p>","DOI":"10.3390\/robotics10010003","type":"journal-article","created":{"date-parts":[[2020,12,25]],"date-time":"2020-12-25T09:30:19Z","timestamp":1608888619000},"page":"3","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Leveraging Graphical User Interface Automation for Generic Robot Programming"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1953-761X","authenticated-orcid":false,"given":"Tudor B.","family":"Ionescu","sequence":"first","affiliation":[{"name":"Human-Machine Interaction Group, Vienna University of Technology, 1040 Vienna, Austria"}]}],"member":"1968","published-online":{"date-parts":[[2020,12,25]]},"reference":[{"key":"ref_1","unstructured":"Biggs, G., and MacDonald, B. (2003, January 10). A survey of robot programming systems. Proceedings of the Australasian Conference on Robotics and Automation, Brisbane, Australia."},{"key":"ref_2","unstructured":"Archibald, C., and Petriu, E. (1993, January 15\u201318). Skills-oriented robot programming. Proceedings of the International Conference on Intelligent Autonomous Systems IAS-3, Pittsburgh, PA, USA."},{"key":"ref_3","unstructured":"Freund, E., and Luedemann-Ravit, B. (October, January 30). A system to automate the generation of program variants for industrial robot applications. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Institute of Electrical and Electronics Engineers (IEEE), New York, NY, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1362","DOI":"10.1109\/LRA.2018.2798300","article-title":"RAZER\u2014A HRI for Visual Task-Level Programming and Intuitive Skill Param-eterization","volume":"3","author":"Steinmetz","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_5","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., and Ng, A.Y. (2009, January 25). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan."},{"key":"ref_6","unstructured":"(2020, November 25). RoboDK. Available online: https:\/\/robodk.com\/."},{"key":"ref_7","unstructured":"(2020, November 25). Siemens Simatic Robot Integration. Available online: https:\/\/new.siemens.com\/global\/en\/products\/automation\/industry-software\/automation-software\/tia-portal\/highlights\/robot-integration.html."},{"key":"ref_8","unstructured":"(2020, November 25). Drag&Bot. Available online: https:\/\/www.dragandbot.com\/."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Garc\u00eda, S., Str\u00fcber, D., Brugali, D., Berger, T., and Pelliccione, P. (2020, January 8\u201313). Robotics software engineering: A perspective from the service robotics domain. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA.","DOI":"10.1145\/3368089.3409743"},{"key":"ref_10","unstructured":"Wienke, J., and Wrede, S. (2017). Results of the survey: Failures in robotics and intelligent systems. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kolak, S., Afzal, A., Le Goues, C., Hilton, M., and Timperley, C.S. (2020, January 22). It takes a village to build a robot: An empirical study of the ros ecosystem. Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), Institute of Electrical and Electronics Engineers (IEEE), New York, NY, USA.","DOI":"10.1109\/ICSME46990.2020.00048"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.jss.2019.02.024","article-title":"The Robot Operating System: Package reuse and community dynamics","volume":"151","author":"Estefo","year":"2018","journal-title":"J. Syst. Softw."},{"key":"ref_13","unstructured":"(2020, November 25). SikuliX. Available online: http:\/\/sikulix.com\/."},{"key":"ref_14","unstructured":"(2020, November 25). PyAutoGUI. Available online: https:\/\/pyautogui.readthedocs.io\/."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1109\/MS.2007.176","article-title":"Ruby on rails","volume":"24","author":"Kirchberg","year":"2007","journal-title":"IEEE Softw."},{"key":"ref_16","unstructured":"(2020, November 25). GRAND GARAGE. Available online: https:\/\/grandgarage.eu\/."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ionescu, T.B. (2019, January 14\u201318). Meet Your Personal Cobot, But Don\u2019t Touch It Just Yet. Proceedings of the 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), New Delhi, India.","DOI":"10.1109\/RO-MAN47096.2020.9223573"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ionescu, T.B., Fr\u00f6hlich, J., and Lachenmayr, M. (2020, January 8\u201311). Improving Safeguards and Functionality in Industrial Collaborative Robot HMIs through GUI Automation. Proceedings of the 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vienna University of Technology (TU Wien), Vienna, Austria.","DOI":"10.1109\/ETFA46521.2020.9211886"},{"key":"ref_19","unstructured":"(2020, November 25). UR Polyscope. Available online: https:\/\/www.universal-robots.com\/."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"38","DOI":"10.3102\/0013189X12463051","article-title":"Computational thinking in K\u201312: A review of the state of the field","volume":"42","author":"Grover","year":"2013","journal-title":"Educ. Res."},{"key":"ref_21","unstructured":"(2020, November 25). Franka Emika Desk. Available online: https:\/\/www.franka.de\/apps\/."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ionescu, T.B. (2019, January 25\u201331). When software development meets the shopfloor: The case of industrial fablabs. Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), Montreal, QC, Canada.","DOI":"10.1109\/ICSE-Companion.2019.00097"},{"key":"ref_23","unstructured":"(2020, November 25). Rethink Robotics Intera. Available online: https:\/\/www.rethinkrobotics.com\/intera\/."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Huang, J., and Cakmak, M. (2017, January 6\u20139). Code3: A system for end-to-end programming of mobile manipulator robots for novices and experts. Proceedings of the 2017 ACM\/IEEE International Conference on HumanRobot Interaction, Vienna, Austria.","DOI":"10.1145\/2909824.3020215"},{"key":"ref_25","unstructured":"Hangl, S., Mennel, A., and Piater, J. (2017). A novel skill-based programming paradigm based on autonomous playing and skill-centric testing. arXiv."},{"key":"ref_26","unstructured":"VDI-Richtlinie 2860 (2010). Montage-und Handhabungstechnik; Handhabungsfunktionen, Handhabungseinrichtungen; Begriffe, Definitionen, Symbole, VDI-Gesellschaft Produktion und Logistik."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Halt, L., Nagele, F., Tenbrock, P., and Pott, A. (2018, January 21\u201325). Intuitive Constraint-Based Robot Programming for Robotic Assembly Tasks. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Institute of Electrical and Elec-tronics Engineers (IEEE), Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8462882"},{"key":"ref_28","unstructured":"(2020, November 25). Dobot. Available online: https:\/\/www.dobot.cc\/."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Weintrop, D., Afzal, A., Salac, J., Francis, P., Li, B., Shepherd, D.C., and Franklin, D. (2018, January 21\u201326). Evaluating CoBlox: A comparative study of robotics programming environments for adult novices. Proceedings of the 2018 Conference on Human Factors in Computing Systems (CHI), Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173940"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Mateo, C., Brunete, A., Gambao, E., and Hernando, M. (2014, January 10\u201312). Hammer: An Android based application for end-user industrial robot programming. Proceedings of the IEEE\/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA), Institute of Electrical and Electronics Engineers (IEEE), New York, NY, USA.","DOI":"10.1109\/MESA.2014.6935597"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Fraser, N. (2015, January 22). Ten things we\u2019ve learned from Blockly. Proceedings of the 2015 IEEE Blocks and Beyond Workshop (Blocks and Beyond), Atlanta, GA, USA.","DOI":"10.1109\/BLOCKS.2015.7369000"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1108\/01439910910994605","article-title":"Technology and applications of ABB RobotStudio","volume":"36","author":"Connolly","year":"2009","journal-title":"Ind. Robot. Int. J."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.procir.2019.03.017","article-title":"A Participatory Programming Model for Democratizing Cobot Technology in Public and Industrial Fablabs","volume":"81","author":"Ionescu","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.jvlc.2016.07.005","article-title":"Designing an API at an appropriate abstraction level for programming social robot applications","volume":"39","author":"Diprose","year":"2017","journal-title":"J. Vis. Lang. Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lewis, R. (2001). Modeling Distributed Control Systems Using IEC 61499\u2014Applying Function Blocks to Distributed Systems, IEEE Pub-lishing.","DOI":"10.1049\/PBCE059E"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"100970","DOI":"10.1016\/j.cola.2020.100970","article-title":"Visual Programming Environments for End-User Development of intelligent and social robots, a Systematic Review","volume":"58","author":"Coronado","year":"2020","journal-title":"J. Comput. Lang."},{"key":"ref_37","unstructured":"(2020, November 22). IGUS Robot Control. Available online: https:\/\/www.igus.eu\/info\/robot-software."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1145\/857076.857078","article-title":"The many faces of publish\/subscribe","volume":"35","author":"Eugster","year":"2003","journal-title":"ACM Comput. Surv."},{"key":"ref_39","unstructured":"(2020, December 17). SMACH. Available online: http:\/\/wiki.ros.org\/smach."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/LRA.2018.2878604","article-title":"Try to Start It! The Challenge of Reusing Code in Robotics Research","volume":"4","author":"Cervera","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_41","unstructured":"(2020, November 25). Artiminds. Available online: https:\/\/www.artiminds.com\/."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Leotta, M., Clerissi, D., Ricca, F., and Spadaro, C. (2013, January 18\u201322). Improving test suites maintainability with the page object pattern: An industrial case study. Proceedings of the IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, Institute of Electrical and Electronics Engineers (IEEE), New York, NY, USA.","DOI":"10.1109\/ICSTW.2013.19"},{"key":"ref_43","unstructured":"Parker, M.H., Kepple, L.R., Sklar, L.R., and Laroche, D.C. (1998). Automated GUI Interface Testing. (5,781,720), U.S. Patent."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Yang, X., Miao, Y., and Zhang, Y. (2011). Model-driven GUI Automation for Efficient Information Exchange between Heterogeneous Electronic Medical Record Systems. Information Systems Development, Springer.","DOI":"10.1007\/978-1-4419-9790-6_65"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1057\/jittc.2016.5","article-title":"Turning robotic process automation into commercial success-case OpusCapita","volume":"6","author":"Asatiani","year":"2016","journal-title":"J. Inf. Technol. Teach. Cases"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Kasper, M., Correll, N., and Yeh, T. (2014, January 4). Abstracting perception and manipulation in end-user robot programming using Sikuli. Proceedings of the IEEE International Conference on Technologies for Practical Robot Applications (TePRA), Woburn, MA, USA.","DOI":"10.1109\/TePRA.2014.6869156"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Polden, J., Pan, Z., Larkin, N., Van Duin, S., and Norrish, J. (2011). Offline Programming for a Complex Welding System using DELMIA Automation. Robotic Welding, Intelligence and Automation, Springer.","DOI":"10.1007\/978-3-642-19959-2_42"},{"key":"ref_48","unstructured":"Huesman, J. (2016). Spaceport Command and Control System User Interface Testing, Technical Report for Launching Commercial and Government Owned Spacecraft."},{"key":"ref_49","unstructured":"Warner, P.C. (2015). Automatic Configuration of Programmable Logic Controller Emulators (No. AFIT-ENG-MS-15-M-024). [Master\u2019s Thesis, Air Force Institute of Technology, Wright-Patterson Graduate School of Engineering, School of Engineering and Management]."},{"key":"ref_50","unstructured":"Corvin, C.M. (2015). A Feasibility Study on the Application of the ScriptGenE Framework as an Anomaly Detection System in In-dustrial Control Systems. [Master\u2019s Thesis, Department of Electrical and Computer Engineering, Air Force Institute of Tech-nology]."},{"key":"ref_51","unstructured":"Gallenstein, J.K. (2017). Integration of the Network and Application Layers of Automatically-Configured Programmable Logic Con-troller Honeypots (No. AFIT-ENG-MS-17-M-029). [Master\u2019s Thesis, Department of Electrical and Computer Engineering, Air Force Institute of Technology]."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Girtz, K., Mullins, B., Rice, M., and Lopez, J. (2016). Practical application layer emulation in industrial control system honeypots. International Conference on Critical Infrastructure Protection, Springer.","DOI":"10.1007\/978-3-319-48737-3_5"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Wang, Z., and Wagner, A. (2019, January 18\u201320). Evaluating a Tactile Approach to Programming Scratch. Proceedings of the 2019 ACM Southeast Conference, Kennesaw, GA, USA.","DOI":"10.1145\/3299815.3314464"},{"key":"ref_54","unstructured":"(2020, November 25). URSim. Available online: https:\/\/www.universal-robots.com\/download\/."},{"key":"ref_55","unstructured":"(2020, November 25). RealVNC. Available online: https:\/\/www.realvnc.com\/."},{"key":"ref_56","unstructured":"(2020, November 25). jQuery Sortable. Available online: https:\/\/jqueryui.com\/sortable\/."},{"key":"ref_57","unstructured":"(2020, November 25). Glumb robot simulator. Available online: https:\/\/github.com\/glumb\/robot-gui."},{"key":"ref_58","first-page":"1008","article-title":"GBB: A Generic Blackboard Development System","volume":"86","author":"Corkill","year":"1986","journal-title":"AAAI"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Yeh, T., Chang, T.H., and Miller, R.C. (2009, January 4\u20137). Sikuli: Using GUI screenshots for search and automation. Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, Victoria, BC, Canada.","DOI":"10.1145\/1622176.1622213"},{"key":"ref_60","unstructured":"Wilson, B.G. (1996). Constructivist Learning Environments: Case Studies in Instructional Design, Educational Technology."},{"key":"ref_61","first-page":"n2","article-title":"Makerspaces for Pedagogical Innovation Processes: How Finnish Comprehensive Schools Create Space for Makers","volume":"24","author":"Jaatinen","year":"2019","journal-title":"Des. Technol. Educ."},{"key":"ref_62","first-page":"3","article-title":"OpenCV","volume":"25","author":"Bradski","year":"2000","journal-title":"Dr. Dobb\u2019s J. Softw. Tools"},{"key":"ref_63","unstructured":"(2020, November 25). ABB Single-Arm Yumi. Available online: https:\/\/new.abb.com\/products\/robotics\/industrial-robots\/irb-14050-single-arm-yumi."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/MRA.2009.934837","article-title":"Component-based robotic engineering (Part I) [Tutorial]","volume":"16","author":"Brugali","year":"2009","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1109\/MRA.2015.2452201","article-title":"Model-driven software engineering in robotics: Models are designed to use the relevant things, thereby reducing the complexity and cost in the field of robotics","volume":"22","author":"Brugali","year":"2015","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Sadik, A.R., Goerick, C., and Muehlig, M. (2019, January 4\u20136). Modeling and Simulation of a Multi-Robot System Architecture. Proceedings of the International Conference on Mechatronics, Robotics and Systems Engineering (MoRSE), Bali, Indonesia.","DOI":"10.1109\/MoRSE48060.2019.8998662"},{"key":"ref_67","unstructured":"Aspridou, M. (2017). Extending BPMN for Modeling Manufacturing Processes. [Master\u2019s Thesis, Business Information Systems, TU Eindhoven, Eindhoven University of Technology]."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/3\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:46:06Z","timestamp":1760179566000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/10\/1\/3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,25]]},"references-count":67,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["robotics10010003"],"URL":"https:\/\/doi.org\/10.3390\/robotics10010003","relation":{},"ISSN":["2218-6581"],"issn-type":[{"type":"electronic","value":"2218-6581"}],"subject":[],"published":{"date-parts":[[2020,12,25]]}}}