{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T15:42:22Z","timestamp":1785512542995,"version":"3.56.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"22","license":[{"start":{"date-parts":[[2022,7,24]],"date-time":"2022-07-24T00:00:00Z","timestamp":1658620800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,24]],"date-time":"2022-07-24T00:00:00Z","timestamp":1658620800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["101000427"],"award-info":[{"award-number":["101000427"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003407","name":"Ministero dell\u2019Istruzione, dell\u2019Universit\u00e0 e della Ricerca","doi-asserted-by":"publisher","award":["Dipartimenti di eccellenza 2018-2022"],"award-info":[{"award-number":["Dipartimenti di eccellenza 2018-2022"]}],"id":[{"id":"10.13039\/501100003407","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Universit\u00e0 degli Studi di Roma La Sapienza"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Several cybersecurity domains, such as ransomware detection, forensics and data analysis, require methods to reliably identify encrypted data fragments. Typically, current approaches employ statistics derived from byte-level distribution, such as entropy estimation, to identify encrypted fragments. However, modern content types use compression techniques which alter data distribution pushing it closer to the uniform distribution. The result is that current approaches exhibit unreliable encryption detection performance when compressed data appear in the dataset. Furthermore, proposed approaches are typically evaluated over few data types and fragment sizes, making it hard to assess their practical applicability. This paper compares existing statistical tests on a large, standardized dataset and shows that current approaches consistently fail to distinguish encrypted and compressed data on both small and large fragment sizes. We address these shortcomings and design <jats:sc>EnCoD<\/jats:sc>, a learning-based classifier which can reliably distinguish compressed and encrypted data. We evaluate <jats:sc>EnCoD<\/jats:sc> on a dataset of 16 different file types and fragment sizes ranging from 512B to 8KB. Our results highlight that <jats:sc>EnCoD<\/jats:sc> outperforms current approaches by a wide margin, with accuracy ranging from <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\sim 82\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mo>\u223c<\/mml:mo>\n                    <mml:mn>82<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> for 512B fragments up to <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\sim 92\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mo>\u223c<\/mml:mo>\n                    <mml:mn>92<\/mml:mn>\n                    <mml:mo>%<\/mml:mo>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> for 8KB data fragments. Moreover, <jats:sc>EnCoD<\/jats:sc> can pinpoint the exact format of a given data fragment, rather than performing only binary classification like previous approaches.<\/jats:p>","DOI":"10.1007\/s00521-022-07586-7","type":"journal-article","created":{"date-parts":[[2022,7,24]],"date-time":"2022-07-24T18:11:27Z","timestamp":1658686287000},"page":"20379-20393","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Reliable detection of compressed and encrypted data"],"prefix":"10.1007","volume":"34","author":[{"given":"Fabio","family":"De Gaspari","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5686-3831","authenticated-orcid":false,"given":"Dorjan","family":"Hitaj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giulio","family":"Pagnotta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorenzo","family":"De Carli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luigi V.","family":"Mancini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,24]]},"reference":[{"key":"7586_CR1","doi-asserted-by":"crossref","unstructured":"Kharraz A, Kirda E (2017) Redemption: real-time protection against ransomware at end-hosts. Research in Attacks, Intrusions, and Defenses","DOI":"10.1007\/978-3-319-66332-6_5"},{"key":"7586_CR2","doi-asserted-by":"crossref","unstructured":"Mehnaz S, Mudgerikar A, Bertino E (2018) Rwguard: a real-time detection system against cryptographic ransomware. Research in Attacks, Intrusions, and Defenses","DOI":"10.1007\/978-3-030-00470-5_6"},{"key":"7586_CR3","doi-asserted-by":"crossref","unstructured":"Continella A et\u00a0al (2016) Shieldfs: a self-healing, ransomware-aware filesystem. In: annual computer security applications conference (ACSAC)","DOI":"10.1145\/2991079.2991110"},{"key":"7586_CR4","doi-asserted-by":"crossref","unstructured":"Kirda E (2017) Unveil: a large-scale, automated approach to detecting ransomware (keynote). In: IEEE international conference on software analysis, evolution and reengineering","DOI":"10.1109\/SANER.2017.7884603"},{"key":"7586_CR5","doi-asserted-by":"publisher","first-page":"S3","DOI":"10.1016\/j.diin.2010.05.002","volume":"7","author":"G Conti","year":"2010","unstructured":"Conti G et al (2010) Automated mapping of large binary objects using primitive fragment type classification. Digit Investig 7:S3\u2013S12","journal-title":"Digit Investig"},{"key":"7586_CR6","doi-asserted-by":"crossref","unstructured":"De\u00a0Carli L, Torres R, Modelo-Howard G, Tongaonkar A, Jha S (2017) Botnet protocol inference in the presence of encrypted traffic. In: IEEE international conference on computer communications","DOI":"10.1109\/INFOCOM.2017.8057064"},{"key":"7586_CR7","doi-asserted-by":"crossref","unstructured":"Dorfinger P, Panholzer G, John W (2011) Entropy estimation for real-time encrypted traffic identification","DOI":"10.1007\/978-3-642-20305-3_14"},{"key":"7586_CR8","doi-asserted-by":"crossref","unstructured":"Fielding R et\u00a0al. (1999) Rfc 2616, hypertext transfer protocol \u2013 http\/1.1 .http:\/\/www.rfc.net\/rfc2616.html","DOI":"10.17487\/rfc2616"},{"issue":"4","key":"7586_CR9","first-page":"89","volume":"1","author":"B Park","year":"2008","unstructured":"Park B et al (2008) Data extraction from damage compressed file for computer forensic purposes. Int J Hybrid Inf Technol 1(4):89\u2013102","journal-title":"Int J Hybrid Inf Technol"},{"key":"7586_CR10","unstructured":"Malhotra P (2007) Detection of encrypted streams for egress monitoring. Master of Science, Iowa State University, Ames. https:\/\/lib.dr.iastate.edu\/rtd\/14632\/"},{"key":"7586_CR11","doi-asserted-by":"crossref","unstructured":"Wang Y, Zhang Z, Guo L, Li S (2011) Using entropy to classify traffic more deeply. In: 2011 IEEE 6th international conference on networking, architecture, and storage pp 45\u201352","DOI":"10.1109\/NAS.2011.18"},{"key":"7586_CR12","doi-asserted-by":"crossref","unstructured":"Mbol F, Robert J-M, Sadighian A (2016) An efficient approach to detect torrentlocker ransomware in computer systems. Cryptol Netw Secur 532\u2013541","DOI":"10.1007\/978-3-319-48965-0_32"},{"key":"7586_CR13","doi-asserted-by":"crossref","unstructured":"Palisse A, Durand A, Le\u00a0Bouder H, LeGuernic C, Lanet J-L (2017) Data aware defense (DaD): towards a generic and practical ransomware countermeasure. Secure IT Syst 10674","DOI":"10.1007\/978-3-319-70290-2_12"},{"key":"7586_CR14","unstructured":"Hahn D, Apthorpe N, Feamster N (2018) Detecting compressed cleartext traffic from consumer internet of things devices 1805:02722"},{"issue":"11","key":"7586_CR15","doi-asserted-by":"publisher","first-page":"2916","DOI":"10.1109\/TIFS.2019.2911156","volume":"14","author":"F Casino","year":"2019","unstructured":"Casino F, Choo K-KR, Patsakis C (2019) HEDGE: efficient traffic classification of encrypted and compressed packets. IEEE Trans Inf Forensics Secur 14(11):2916\u20132926","journal-title":"IEEE Trans Inf Forensics Secur"},{"issue":"1\u20132","key":"7586_CR16","doi-asserted-by":"publisher","first-page":"603","DOI":"10.1007\/s11042-019-08088-w","volume":"79","author":"P Choudhury","year":"2020","unstructured":"Choudhury P, Kumar KRP, Nandi S, Athithan G (2020) An empirical approach towards characterization of encrypted and unencrypted VoIP traffic. Multimed Tools Appl 79(1\u20132):603\u2013631","journal-title":"Multimed Tools Appl"},{"key":"7586_CR17","doi-asserted-by":"crossref","unstructured":"De\u00a0Gaspari F, Hitaj D, Pagnotta G, De Carli L, Mancini L V (2020) The naked sun: malicious cooperation between benign-looking processes. In: 18th international conference on applied cryptography and network security","DOI":"10.1007\/978-3-030-57878-7_13"},{"key":"7586_CR18","doi-asserted-by":"crossref","unstructured":"De\u00a0Gaspari F, Hitaj D, Pagnotta G, De Carli L, Mancini L V (2022) Evading behavioral classifiers: a comprehensive analysis on evading ransomware detection techniques. Neural Comput Appl","DOI":"10.1007\/s00521-022-07096-6"},{"key":"7586_CR19","unstructured":"Lee H, Ge R, Ma T, Risteski A, Arora S (2017) On the ability of neural nets to express distributions. In: Proceedings of the 30th conference on learning theory, COLT"},{"key":"7586_CR20","doi-asserted-by":"crossref","unstructured":"Pagnotta G, Hitaj D, De\u00a0Gaspari F, Mancini L V (2022) Passflow: guessing passwords with generative flows. In: Proceedings of the 52nd Annual IEEE\/IFIP international conference on dependable systems and networks (DSN\u201922)","DOI":"10.1109\/DSN53405.2022.00035"},{"key":"7586_CR21","doi-asserted-by":"crossref","unstructured":"De\u00a0Gaspari F, Hitaj D, Pagnotta G, De\u00a0Carli L, Mancini L V (2020) Encod: distinguishing compressed and encrypted file fragments. In: international conference on network and system security pp 42\u201362","DOI":"10.1007\/978-3-030-65745-1_3"},{"key":"7586_CR22","unstructured":"Atlanta spent \\$2.6m to recover from a \\$52,000 ransomware scare. https:\/\/www.wired.com\/story\/atlanta-spent-26m-recover-from-ransomware-scare\/ (2018)"},{"key":"7586_CR23","doi-asserted-by":"crossref","unstructured":"Wannacry cyber attack cost the nhs \u00a392m as 19,000 appointments cancelled. https:\/\/www.telegraph.co.uk\/technology\/2018\/10\/11\/wannacry-cyber-attack-cost-nhs-92m-19000-appointments-cancelled\/ (2018)","DOI":"10.1016\/S1361-3723(18)30102-7"},{"key":"7586_CR24","unstructured":"Ransomware attacks grow, crippling cities and businesses. https:\/\/www.nytimes.com\/2020\/02\/09\/technology\/ransomware-attacks.html (2020)"},{"key":"7586_CR25","unstructured":"Walls RJ, Learned-Miller E, Levine BN (2011) Forensic triage for mobile phones with DEC0DE. In: USENIX Security Symposium"},{"key":"7586_CR26","doi-asserted-by":"crossref","unstructured":"Wallace G K (1992) The jpeg still picture compression standard. IEEE Trans Consum Electron","DOI":"10.1109\/30.125072"},{"key":"7586_CR27","unstructured":"Rukhin A et\u00a0al. (2010) A statistical test suite for random and pseudorandom number generators for cryptographic applications. Special Publication 800-22r1a, NIST"},{"key":"7586_CR28","unstructured":"Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. Soc Artif Intell Stat (AISTATS)"},{"key":"7586_CR29","doi-asserted-by":"crossref","unstructured":"LeCun Y, Bottou L, Orr G B, M\u00fcller K-R (1998) Efficient backprop. Neural Networks: Tricks of the Trade","DOI":"10.1007\/3-540-49430-8_2"},{"key":"7586_CR30","doi-asserted-by":"crossref","unstructured":"Trottier L, Giguere P, Chaib-draa B (2017) Parametric exponential linear unit for deep convolutional neural networks. In: 2017 16th IEEE international conference on machine learning and applications (ICMLA)","DOI":"10.1109\/ICMLA.2017.00038"},{"key":"7586_CR31","unstructured":"Klambauer G, Unterthiner T, Mayr A, Hochreiter S (2017) Self-normalizing neural networks. CoRR abs\/1706.02515"},{"key":"7586_CR32","doi-asserted-by":"crossref","unstructured":"Piskozub M, De\u00a0Gaspari F, Barr-Smith F, Mancini L, Martinovic I (2021) Malphase: fine-grained malware detection using network flow data. In: Proceedings of the 2021 ACM on asia conference on computer and communications security","DOI":"10.1145\/3433210.3453101"},{"key":"7586_CR33","doi-asserted-by":"crossref","unstructured":"Qi Y, Wang Y, Zheng X, Wu Z (2014) Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)","DOI":"10.1109\/ICASSP.2014.6854900"},{"key":"7586_CR34","doi-asserted-by":"crossref","unstructured":"Nguyen A, Yosinski J, Clune J (2015) Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR) pp 427\u2013436","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"7586_CR35","doi-asserted-by":"crossref","unstructured":"Mamun MS I, Ghorbani A A, Stakhanova N (2016) An entropy based encrypted traffic classifier. Inf Commun Secur 282\u2013294","DOI":"10.1007\/978-3-319-29814-6_23"},{"key":"7586_CR36","doi-asserted-by":"crossref","unstructured":"Zhang H, Papadopoulos C, Massey D (2013) Detecting encrypted botnet traffic. In: 2013 Proceedings IEEE INFOCOM pp 3453\u20131358","DOI":"10.1109\/INFCOM.2013.6567180"},{"key":"7586_CR37","unstructured":"Wang R, Shoshitaishvili Y, Kruegel C, Vigna G (2013) Steal this movie - automatically bypassing drm protection in streaming media services. In: 22nd USENIX Security Symposium"},{"issue":"1","key":"7586_CR38","first-page":"6","volume":"3","author":"N Ameeno","year":"2019","unstructured":"Ameeno N, Sherry K, Gagneja K (2019) Using machine learning to detect the file compression or encryption. Amity J Comput Sci 3(1):6","journal-title":"Amity J Comput Sci"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07586-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-022-07586-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07586-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T21:03:09Z","timestamp":1666299789000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-022-07586-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,24]]},"references-count":38,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["7586"],"URL":"https:\/\/doi.org\/10.1007\/s00521-022-07586-7","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,24]]},"assertion":[{"value":"8 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}