{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T10:59:57Z","timestamp":1777805997577,"version":"3.51.4"},"reference-count":40,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2016,6,2]],"date-time":"2016-06-02T00:00:00Z","timestamp":1464825600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Computer Security"],"published-print":{"date-parts":[[2016,11,8]]},"abstract":"<jats:p>Many network security systems analyze large scale data collected from multiple collaborating domains or aggregated network vantage points. Scale is clearly beneficial for these systems, however it also makes them difficult to design and test. Large scale data sets can be difficult to acquire and may not contain important meta-information (e.g. ground truth). Further, their limited availability can make it extremely difficult to understand how well experimental results would reproduce in different conditions, or at different networks. In this article, we discuss using simulation to overcome these challenges. We present an augmented version of LESS, our recently proposed agent based simulator for evaluating large scale network security systems. LESS uses publicly available data sets and high level parameters to generate synthetic traffic that models large scale, multi-network scenarios. Essentially, LESS allows researchers to \u201cscale up\u201d the data and statistics about networks and attacks that they have access to, so that they can be used to test large scale network security systems. Researchers can also tune LESS\u2019s high level parameters to better understand the sensitivities of their systems, and the reproducibility of their results. The version of LESS that we discuss in this article is extended to allow researchers to study an additional factor of system performance related to reproducibility: deployment location; by modeling the global Internet topology at the Autonomous System level. We demonstrate the applicability and benefits of LESS by tuning it with publicly available traces and then using generated records to reproduce and extend results from several recently proposed large scale security systems. In new experiments, we use LESS to study how deployment location affects large scale security systems. Our results demonstrate that LESS can evoke realistic performance from these systems with minimal tuning and provide insight into the network and topological factors that may affect the reproducibility of their evaluations.<\/jats:p>","DOI":"10.3233\/jcs-160553","type":"journal-article","created":{"date-parts":[[2016,11,11]],"date-time":"2016-11-11T10:41:00Z","timestamp":1478860860000},"page":"645-665","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Exploring large scale security system reproducibility with the LESS simulator"],"prefix":"10.1177","volume":"24","author":[{"given":"John","family":"Sonchack","sequence":"first","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA 19104, USA. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adam J.","family":"Aviv","sequence":"additional","affiliation":[{"name":"United States Naval Academy, Annapolis, MD 21402, USA. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2016,6,2]]},"reference":[{"key":"ref001","unstructured":"Argus, Audit records generation and utilization system, available at: http:\/\/qosient.com\/argus\/."},{"key":"ref002","unstructured":"A.J.Aviv and A.Haeberlen, Challenges in experimenting with botnet detection systems, in: Proceedings of the 4th Conference On Cyber Security Experimentation And Test \u2013 CSET\u201911, ACM, 2011."},{"key":"ref003","doi-asserted-by":"publisher","DOI":"10.1126\/science.286.5439.509"},{"key":"ref004","doi-asserted-by":"crossref","unstructured":"V.D.Blondel, J.L.Guillaume, R.Lambiotte and E.Lefebvre, Fast unfolding of communities in large networks, Journal of Statistical Mechanics: Theory and Experiment 2008(10) (2008), P10008. doi:10.1088\/1742-5468\/2008\/10\/P10008.","DOI":"10.1088\/1742-5468\/2008\/10\/P10008"},{"key":"ref005","doi-asserted-by":"crossref","unstructured":"N.Boggs, S.Hiremagalore, A.Stavrou and S.J.Stolfo, Cross-domain collaborative anomaly detection: So far yet so close, in: RAID, Springer, 2011, pp.\u00a0142\u2013160.","DOI":"10.1007\/978-3-642-23644-0_8"},{"key":"ref006","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.082080899"},{"key":"ref007","unstructured":"Caida data overview, available at: http:\/\/www.caida.org\/data\/overview\/."},{"key":"ref008","doi-asserted-by":"crossref","unstructured":"J.Cao, W.S.Cleveland, Y.Gao, K.Jeffay, F.D.Smith and M.Weigle, Stochastic models for generating synthetic http source traffic, in: The 23rd Annual IEEE International Conference on Computer Communications \u2013 IEEE INFOCOM 2004, Vol. 3, IEEE, 2004, pp.\u00a01546\u20131557.","DOI":"10.1109\/INFCOM.2004.1354568"},{"key":"ref009","doi-asserted-by":"crossref","unstructured":"G.Chen and R.S.Gray, Simulating non-scanning worms on peer-to-peer networks, in: Proceedings of the 1st International Conference on Scalable Information Systems \u2013 InfoScale\u201906, ACM, 2006, Article no.\u00a029. doi:10.1145\/1146847.1146876.","DOI":"10.1145\/1146847.1146876"},{"key":"ref010","doi-asserted-by":"crossref","unstructured":"B.Coskun, S.Dietrich and N.Memon, Friends of an enemy: Identifying local members of peer-to-peer botnets using mutual contacts, in: Proceedings of the 26th Annual Computer Security Applications Conference, 2010.","DOI":"10.1145\/1920261.1920283"},{"key":"ref011","unstructured":"X.Dimitropoulos, D.Krioukov, G.Riley et al., Revealing the autonomous system taxonomy: The machine learning approach, 2006, available at: arXiv:cs\/0604015."},{"key":"ref012","doi-asserted-by":"publisher","DOI":"10.1080\/00018730110112519"},{"key":"ref013","unstructured":"Dshield, available at: http:\/\/www.dshield.org\/."},{"key":"ref014","doi-asserted-by":"publisher","DOI":"10.5486\/PMD.1959.6.3-4.12"},{"key":"ref015","doi-asserted-by":"publisher","DOI":"10.1109\/90.944338"},{"key":"ref016","unstructured":"J.B.Grizzard, V.Sharma, C.Nunnery, B.B.Kang and D.Dagon, Peer-to-peer botnets: Overview and case study, in: Proceedings of the First Conference on First Workshop on Hot Topics in Understanding Botnets \u2013 HotBots\u201907, 2007, pp.\u00a01\u20138, Article no. 1."},{"key":"ref017","unstructured":"A.Hagberg, P.Swart and D.S.Chult, Exploring network structure, dynamics, and function using networkX, Technical report, Los Alamos National Laboratory (LANL), 2008."},{"key":"ref018","doi-asserted-by":"crossref","unstructured":"J.Hawkinson and T.Bates, Guidelines for creation, selection, and registration of an autonomous system (AS), 1996.","DOI":"10.17487\/rfc1930"},{"key":"ref019","first-page":"80","volume":"1","author":"Hutchins E.M.","year":"2011","journal-title":"Leading Issues in Information Warfare & Security Research"},{"key":"ref020","doi-asserted-by":"crossref","unstructured":"S.Katti, B.Krishnamurthy and D.Katabi, Collaborating against common enemies, in: ACM IMC, 2005.","DOI":"10.1145\/1330107.1330151"},{"key":"ref021","doi-asserted-by":"crossref","unstructured":"V.Konda and J.Kaur, RAPID: Shrinking the congestion-control timescale, in: The 28th Annual IEEE International Conference on Computer Communications \u2013 IEEE INFOCOM 2009, IEEE, 2009, pp.\u00a01\u20139. doi:10.1109\/INFCOM.2009.5061900.","DOI":"10.1109\/INFCOM.2009.5061900"},{"key":"ref022","doi-asserted-by":"crossref","unstructured":"B.Lantz, B.Heller and N.McKeown, A network in a laptop: Rapid prototyping for software-defined networks, in: Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks \u2013 HotNets-IX, ACM, 2010, Article no.\u00a019. doi:10.1145\/1868447.1868466.","DOI":"10.1145\/1868447.1868466"},{"key":"ref023","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2008.4483668"},{"key":"ref024","doi-asserted-by":"crossref","unstructured":"D.Moore, C.Shannon et al., Code-red: A case study on the spread and victims of an Internet worm, in: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurment, ACM, 2002, pp.\u00a0273\u2013284. doi:10.1145\/637201.637244.","DOI":"10.1145\/637201.637244"},{"key":"ref025","doi-asserted-by":"crossref","unstructured":"D.Moore, C.Shannon, G.M.Voelker and S.Savage, Internet quarantine: Requirements for containing self-propagating code, in: The 22nd Annual IEEE International Conference on Computer Communications \u2013 IEEE INFOCOM 2003, Vol. 3, IEEE, 2003, pp.\u00a01901\u20131910.","DOI":"10.1109\/INFCOM.2003.1209212"},{"key":"ref026","unstructured":"Plot digitizer, available at: http:\/\/plotdigitizer.sourceforge.net\/."},{"key":"ref027","unstructured":"N.Provos, D.McNamee, P.Mavrommatis, K.Wang, N.Modadugu et al., The ghost in the browser: Analysis of web-based malware, in: Proceedings of the First Conference on First Workshop on Hot Topics in Understanding Botnets \u2013 HotBots\u201907, ACM, 2007, pp.\u00a01\u20139, Article no. 4."},{"key":"ref028","doi-asserted-by":"crossref","unstructured":"G.F.Riley, The Georgia tech network simulator, in: Proceedings of the ACM SIGCOMM MoMeTools Workshop, ACM, 2003, pp.\u00a05\u201312. doi:10.1145\/944773.944775.","DOI":"10.1145\/944773.944775"},{"key":"ref029","doi-asserted-by":"crossref","unstructured":"B.D.Ripley, Stochastic Simulation, Wiley Series in Probability and Statistics, Vol. 316, Wiley, 1987.","DOI":"10.1002\/9780470316726"},{"key":"ref030","doi-asserted-by":"crossref","unstructured":"L.M.Rossey, R.K.Cunningham, D.J.Fried, J.C.Rabek, R.P.Lippmann, J.W.Haines and M.A.Zissman, LARIAT: Lincoln adaptable real-time information assurance testbed, in: 2002 IEEE Aerospace Conference Proceedings, Vol. 6, IEEE, 2002, pp.\u00a02671\u20132682. doi:10.1109\/AERO.2002.1036140.","DOI":"10.1109\/AERO.2002.1036158"},{"key":"ref031","doi-asserted-by":"crossref","unstructured":"J.Sommers, V.Yegneswaran and P.Barford, Recent advances in network intrusion detection system tuning, in: 40th Annual CISS, IEEE, 2006, pp.\u00a01490\u20131495.","DOI":"10.1109\/CISS.2006.286375"},{"key":"ref032","doi-asserted-by":"crossref","unstructured":"J.Sonchack and A.J.Aviv, Less is more: Host-agent based simulator for large-scale evaluation of security systems, in: Computer Security \u2013 ESORICS 2014, Springer, 2014, pp.\u00a0365\u2013382.","DOI":"10.1007\/978-3-319-11212-1_21"},{"key":"ref033","unstructured":"J.Sonchack, A.J.Aviv and J.M.Smith, Bridging the data gap: Data related challenges in evaluating large scale collaborative security systems, in: 6th Workshop on Cyber Security Experitmentation and Testing, 2013."},{"key":"ref034","doi-asserted-by":"publisher","DOI":"10.1017\/S0963548399003867"},{"key":"ref035","unstructured":"G.Tan, M.Poletto, J.V.Guttag and M.F.Kaashoek, Role classification of hosts within enterprise networks based on connection patterns, in: Proceedings of the Annual Conference On Usenix Annual Technical Conference \u2013 ATEC\u201903, ACM, 2003, pp.\u00a015\u201328."},{"key":"ref036","unstructured":"The caida as relationships dataset, available at: http:\/\/www.caida.org\/data\/as-relationships\/."},{"key":"ref037","doi-asserted-by":"crossref","unstructured":"A.Wagner and B.Plattner, Entropy based worm and anomaly detection in fast IP networks, in: 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise \u2013 WETICE\u201905, IEEE, 2005, pp.\u00a0172\u2013177. doi:10.1109\/WETICE.2005.35.","DOI":"10.1109\/WETICE.2005.35"},{"key":"ref038","doi-asserted-by":"publisher","DOI":"10.1145\/1140086.1140094"},{"key":"ref039","doi-asserted-by":"crossref","unstructured":"G.Xie, M.Iliofotou, R.Keralapura, M.Faloutsos and A.Nucci, Subflow: Towards practical flow-level traffic classification, in: The 31st Annual IEEE International Conference on Computer Communications \u2013 IEEE INFOCOM 2012, IEEE, 2012, pp.\u00a02541\u20132545. doi:10.1109\/INFCOM.2012.6195649.","DOI":"10.1109\/INFCOM.2012.6195649"},{"key":"ref040","unstructured":"J.Zhang, P.Porras and J.Ullrich, Highly predictive blacklisting, in: Proceedings of the 17th Conference On Security Symposium \u2013 SS\u201908, ACM, 2008, pp.\u00a0107\u2013122."}],"container-title":["Journal of Computer Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JCS-160553","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JCS-160553","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JCS-160553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T20:44:59Z","timestamp":1777495499000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JCS-160553"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,6,2]]},"references-count":40,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2016,11,8]]}},"alternative-id":["10.3233\/JCS-160553"],"URL":"https:\/\/doi.org\/10.3233\/jcs-160553","relation":{},"ISSN":["0926-227X","1875-8924"],"issn-type":[{"value":"0926-227X","type":"print"},{"value":"1875-8924","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,6,2]]}}}