{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:08:10Z","timestamp":1750306090741,"version":"3.41.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2017,5,25]],"date-time":"2017-05-25T00:00:00Z","timestamp":1495670400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Auton. Adapt. Syst."],"published-print":{"date-parts":[[2017,6,30]]},"abstract":"<jats:p>Agent-based modeling is a paradigm of modeling dynamic systems of interacting agents that are individually governed by specified behavioral rules. Training a model of such agents to produce an emergent behavior by specification of the emergent (as opposed to agent) behavior is easier from a demonstration perspective. While many approaches involve manual behavior specification via code or reliance on a defined taxonomy of possible behaviors, the meta-modeling framework in Miner [2010] generates mapping functions between agent-level parameters and swarm-level parameters, which are re-usable once generated. This work builds on that framework by integrating demonstration by image or video. The demonstrator specifies spatial motion of the agents over time and retrieves agent-level parameters required to execute that motion. The framework, at its core, uses computationally cheap image-processing algorithms. Our work is tested with a combination of primitive visual feature extraction methods (contour area and shape) and features generated using a pre-trained deep neural network in different stages of image featurization. The framework is also evaluated for its potential using complex visual features for all image featurization stages. Experimental results show significant coherence between demonstrated behavior and predicted behavior based on estimated agent-level parameters specific to the spatial arrangement of agents.<\/jats:p>","DOI":"10.1145\/3084541","type":"journal-article","created":{"date-parts":[[2017,5,25]],"date-time":"2017-05-25T16:16:45Z","timestamp":1495729005000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Feature Construction for Controlling Swarms by Visual Demonstration"],"prefix":"10.1145","volume":"12","author":[{"given":"Karan K.","family":"Budhraja","sequence":"first","affiliation":[{"name":"University of Maryland, Baltimore County, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Winder","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Oates","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,5,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2700318"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-012-0075-2"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 139--146","author":"Brambilla Manuele","year":"2012","unstructured":"Manuele Brambilla , Carlo Pinciroli , Mauro Birattari , and Marco Dorigo . 2012 . Property-driven design for swarm robotics . In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 139--146 . Manuele Brambilla, Carlo Pinciroli, Mauro Birattari, and Marco Dorigo. 2012. Property-driven design for swarm robotics. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 139--146."},{"volume-title":"Neuroevolution-Based Inverse Reinforcement Learning. Master\u2019s thesis","author":"Budhraja Karan K.","key":"e_1_2_1_4_1","unstructured":"Karan K. Budhraja . 2015. Neuroevolution-Based Inverse Reinforcement Learning. Master\u2019s thesis . University of Maryland , Baltimore County . Karan K. Budhraja. 2015. Neuroevolution-Based Inverse Reinforcement Learning. Master\u2019s thesis. University of Maryland, Baltimore County."},{"volume-title":"Proceedings of the 2016 IEEE 10th International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201916)","author":"Karan","key":"e_1_2_1_5_1","unstructured":"Karan K. Budhraja and Tim Oates. 2016a. Controlling swarms by visual demonstration . In Proceedings of the 2016 IEEE 10th International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201916) , IEEE, 1--10. Karan K. Budhraja and Tim Oates. 2016a. Controlling swarms by visual demonstration. In Proceedings of the 2016 IEEE 10th International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201916), IEEE, 1--10."},{"key":"e_1_2_1_6_1","volume-title":"Budhraja and Tim Oates","author":"Karan","year":"2016","unstructured":"Karan K. Budhraja and Tim Oates . 2016 b. Neuroevolution-based inverse reinforcement learning. arXiv:1608.02971 (2016). Karan K. Budhraja and Tim Oates. 2016b. Neuroevolution-based inverse reinforcement learning. arXiv:1608.02971 (2016)."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the AAAI Fall Symposium on Human Control of Bioinspired Swarms. 21--25","author":"Coppin Gilles","year":"2012","unstructured":"Gilles Coppin and Fran\u00e7ois Legras . 2012 . Controlling swarms of unmanned vehicles through user-centered commands . In Proceedings of the AAAI Fall Symposium on Human Control of Bioinspired Swarms. 21--25 . Gilles Coppin and Fran\u00e7ois Legras. 2012. Controlling swarms of unmanned vehicles through user-centered commands. In Proceedings of the AAAI Fall Symposium on Human Control of Bioinspired Swarms. 21--25."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","volume":"1","author":"D\u2019Ambrosio David B.","unstructured":"David B. D\u2019Ambrosio , Joel Lehman , Sebastian Risi , and Kenneth O. Stanley . 2010. Evolving policy geometry for scalable multiagent learning . In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems : Volume 1 . International Foundation for Autonomous Agents and Multiagent Systems, 731--738. David B. D\u2019Ambrosio, Joel Lehman, Sebastian Risi, and Kenneth O. Stanley. 2010. Evolving policy geometry for scalable multiagent learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 731--738."},{"key":"e_1_2_1_9_1","volume-title":"Otdelenie Matematicheskii i Estestvennyka Nauk 7, 793-800","author":"Delaunay Boris","year":"1934","unstructured":"Boris Delaunay . 1934. Sur la sphere vide. Izv. Akad. Nauk SSSR , Otdelenie Matematicheskii i Estestvennyka Nauk 7, 793-800 ( 1934 ), 1--2. Boris Delaunay. 1934. Sur la sphere vide. Izv. Akad. Nauk SSSR, Otdelenie Matematicheskii i Estestvennyka Nauk 7, 793-800 (1934), 1--2."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACC.2015.7170761"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-014-0092-4"},{"volume-title":"RoboCup 2014","author":"Freelan David","key":"e_1_2_1_13_1","unstructured":"David Freelan , Drew Wicke , Keith Sullivan , and Sean Luke . 2014. Towards rapid multi-robot learning from demonstration at the robocup competition . In RoboCup 2014 : Robot World Cup XVIII. Springer , 369--382. David Freelan, Drew Wicke, Keith Sullivan, and Sean Luke. 2014. Towards rapid multi-robot learning from demonstration at the robocup competition. In RoboCup 2014: Robot World Cup XVIII. Springer, 369--382."},{"key":"e_1_2_1_14_1","volume-title":"Deep residual learning for image recognition. arXiv:1512.03385","author":"He Kaiming","year":"2015","unstructured":"Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . 2015. Deep residual learning for image recognition. arXiv:1512.03385 ( 2015 ). Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. arXiv:1512.03385 (2015)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_1_16_1","volume-title":"Kingma and Max Welling","author":"Diederik","year":"2013","unstructured":"Diederik P. Kingma and Max Welling . 2013 . Auto-encoding variational bayes. arXiv:1312.6114 (2013). Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114 (2013)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-013-0079-6"},{"volume-title":"Rule abstraction: Understanding emergent behavior in swarm systems. PhD Proposal","author":"Miner Don","key":"e_1_2_1_20_1","unstructured":"Don Miner . 2009. Rule abstraction: Understanding emergent behavior in swarm systems. PhD Proposal , University of Maryland , Baltimore County . Don Miner. 2009. Rule abstraction: Understanding emergent behavior in swarm systems. PhD Proposal, University of Maryland, Baltimore County."},{"key":"e_1_2_1_22_1","series-title":"Proceedings of the 2009 AAAI Fall Symposium Series.","volume-title":"Predicting and controlling system-level parameters of multi-agent systems","author":"Miner Don","unstructured":"Don Miner and others. 2009. Predicting and controlling system-level parameters of multi-agent systems . In Proceedings of the 2009 AAAI Fall Symposium Series. Don Miner and others. 2009. Predicting and controlling system-level parameters of multi-agent systems. In Proceedings of the 2009 AAAI Fall Symposium Series."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems. Citeseer.","author":"Miner Don","year":"2008","unstructured":"Don Miner and Marie desJardins . 2008 . Learning abstract properties of swarm systems . In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems. Citeseer. Don Miner and Marie desJardins. 2008. Learning abstract properties of swarm systems. In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems. Citeseer."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/SASO.2009.8"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/1620270.1620371"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.1994.576366"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/0031-3203(95)00067-4"},{"key":"e_1_2_1_28_1","first-page":"411","article-title":"Running experiments on amazon mechanical turk","volume":"5","author":"Paolacci Gabriele","year":"2010","unstructured":"Gabriele Paolacci , Jesse Chandler , and Panagiotis G. Ipeirotis . 2010 . Running experiments on amazon mechanical turk . Judg . Dec. Mak. 5 , 5 (2010), 411 -- 419 . Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judg. Dec. Mak. 5, 5 (2010), 411--419.","journal-title":"Judg"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2695664.2695913"},{"key":"e_1_2_1_30_1","volume-title":"remote-access swarm robotics research on the robotarium. arXiv:1604.00640","author":"Pickem Daniel","year":"2016","unstructured":"Daniel Pickem , Li Wang , Paul Glotfelter , Yancy Diaz-Mercado , Mark Mote , Aaron Ames , Eric Feron , and Magnus Egerstedt . 2016. Safe , remote-access swarm robotics research on the robotarium. arXiv:1604.00640 ( 2016 ). Daniel Pickem, Li Wang, Paul Glotfelter, Yancy Diaz-Mercado, Mark Mote, Aaron Ames, Eric Feron, and Magnus Egerstedt. 2016. Safe, remote-access swarm robotics research on the robotarium. arXiv:1604.00640 (2016)."},{"volume-title":"Information Processing in Cells and Tissues","author":"Reeder John","key":"e_1_2_1_31_1","unstructured":"John Reeder . 2015. Team search tactics through multi-agent HyperNEAT . In Information Processing in Cells and Tissues . Springer , 75--89. John Reeder. 2015. Team search tactics through multi-agent HyperNEAT. In Information Processing in Cells and Tissues. Springer, 75--89."},{"key":"e_1_2_1_32_1","volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman . 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 ( 2014 ). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_33_1","volume-title":"Inverse reinforcement learning in swarm systems. arXiv:1602.05450","author":"\u0160o\u0161i\u0107 Adrian","year":"2016","unstructured":"Adrian \u0160o\u0161i\u0107 , Wasiur R. KhudaBukhsh , Abdelhak M. Zoubir , and Heinz Koeppl . 2016. Inverse reinforcement learning in swarm systems. arXiv:1602.05450 ( 2016 ). Adrian \u0160o\u0161i\u0107, Wasiur R. KhudaBukhsh, Abdelhak M. Zoubir, and Heinz Koeppl. 2016. Inverse reinforcement learning in swarm systems. arXiv:1602.05450 (2016)."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 197--204","author":"Sullivan Keith","year":"2012","unstructured":"Keith Sullivan and Sean Luke . 2012 . Learning from demonstration with swarm hierarchies . In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 197--204 . Keith Sullivan and Sean Luke. 2012. Learning from demonstration with swarm hierarchies. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 197--204."},{"key":"e_1_2_1_35_1","volume-title":"Residual networks are exponential ensembles of relatively shallow networks. arXiv:1605.06431","author":"Veit Andreas","year":"2016","unstructured":"Andreas Veit , Michael Wilber , and Serge Belongie . 2016. Residual networks are exponential ensembles of relatively shallow networks. arXiv:1605.06431 ( 2016 ). Andreas Veit, Michael Wilber, and Serge Belongie. 2016. Residual networks are exponential ensembles of relatively shallow networks. arXiv:1605.06431 (2016)."},{"volume-title":"NetLogo flocking model","author":"Wilensky Uri","key":"e_1_2_1_36_1","unstructured":"Uri Wilensky . 1998. NetLogo flocking model . Center for Connected Learning and Computer-Based Modeling , Northwestern University , Evanston, IL. Uri Wilensky. 1998. NetLogo flocking model. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL."},{"key":"e_1_2_1_37_1","unstructured":"Uri Wilensky and I. Evanston. 1999. NetLogo: Center for connected learning and computer-based modeling. Northwestern University Evanston IL (1999) 49--52.  Uri Wilensky and I. Evanston. 1999. NetLogo: Center for connected learning and computer-based modeling. Northwestern University Evanston IL (1999) 49--52."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/SASO.2009.12"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems. 802--810","author":"Xingjian SHI","year":"2015","unstructured":"SHI Xingjian , Zhourong Chen , Hao Wang , Dit-Yan Yeung , Wai-kin Wong, and Wang-chun Woo. 2015 . Convolutional LSTM network: A machine learning approach for precipitation nowcasting . In Proceedings of the Conference on Advances in Neural Information Processing Systems. 802--810 . SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 802--810."}],"container-title":["ACM Transactions on Autonomous and Adaptive Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3084541","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3084541","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:22Z","timestamp":1750217422000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3084541"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,25]]},"references-count":37,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2017,6,30]]}},"alternative-id":["10.1145\/3084541"],"URL":"https:\/\/doi.org\/10.1145\/3084541","relation":{},"ISSN":["1556-4665","1556-4703"],"issn-type":[{"type":"print","value":"1556-4665"},{"type":"electronic","value":"1556-4703"}],"subject":[],"published":{"date-parts":[[2017,5,25]]},"assertion":[{"value":"2016-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-05-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}