{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T06:11:26Z","timestamp":1775283086536,"version":"3.50.1"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2007,7,1]],"date-time":"2007-07-01T00:00:00Z","timestamp":1183248000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2007,7]]},"abstract":"<jats:p>Eleven variants of six widely used open-source spam filters are tested on a chronological sequence of 49086 e-mail messages received by an individual from August 2003 through March 2004. Our approach differs from those previously reported in that the test set is large, comprises uncensored raw messages, and is presented to each filter sequentially with incremental feedback. Misclassification rates and Receiver Operating Characteristic Curve measurements are reported, with statistical confidence intervals. Quantitative results indicate that content-based filters can eliminate 98% of spam while incurring 0.1% legitimate email loss. Qualitative results indicate that the risk of loss depends on the nature of the message, and that messages likely to be lost may be those that are less critical. More generally, our methodology has been encapsulated in a free software toolkit, which may used to conduct similar experiments.<\/jats:p>","DOI":"10.1145\/1247715.1247717","type":"journal-article","created":{"date-parts":[[2007,9,14]],"date-time":"2007-09-14T13:44:55Z","timestamp":1189777495000},"page":"11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":70,"title":["Online supervised spam filter evaluation"],"prefix":"10.1145","volume":"25","author":[{"given":"Gordon V.","family":"Cormack","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Thomas R.","family":"Lynam","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]}],"member":"320","published-online":{"date-parts":[[2007,7]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Agresti A. 1996. An Introduction to Categorical Data Analysis. Wiley New York.  Agresti A. 1996. An Introduction to Categorical Data Analysis. Wiley New York."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Workshop on Machine Learning in the New Information Age, 11th European Conference on Machine Learning (ECML","author":"Androutsopoulos I.","year":"2000"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Workshop on Machine Learning and Textual Information Access, 4th European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD","author":"Androutsopoulos I.","year":"2000"},{"key":"e_1_2_1_4_1","volume-title":"Tech. Rep. 2004\/2, NCSR Demokritos.","author":"Androutsopoulos I.","year":"2004"},{"key":"e_1_2_1_5_1","unstructured":"Burton B. 2002. Spamprobe---A fast Bayesian spam filter. http:\/\/spamprobe.sourceforge.net.  Burton B. 2002. Spamprobe---A fast Bayesian spam filter. http:\/\/spamprobe.sourceforge.net."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 15th Text REtrieval Conference (TREC-2005)","author":"Cormack G. V.","year":"2006"},{"key":"e_1_2_1_7_1","volume-title":"CEAS 2006---The 3rd Conference on Email and Anti-Spam. Mountain View.","author":"Cormack G. V."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of CEAS 2005---The Second Conference on Email and Anti-Spam.","author":"Cormack G. V."},{"key":"e_1_2_1_9_1","volume-title":"TREC 2005 Spam Track Overview. In Fourteenth Text REtrieval Conference (TREC-2005)","author":"Cormack G. V."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/72.788645"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/980972.980990"},{"key":"e_1_2_1_12_1","unstructured":"Fawcett T. 2003b. ROC graphs: Notes and practical considerations for data mining researchers. Tech. rep. HP Laboratories.  Fawcett T. 2003b. ROC graphs: Notes and practical considerations for data mining researchers. Tech. rep. HP Laboratories."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0305004100009580"},{"key":"e_1_2_1_14_1","unstructured":"Flach P. A. 2004. The many faces of roc analysis in machine learning. http:\/\/www.cs.bris.ac.uk\/~flach\/ICML04tutorial\/.  Flach P. A. 2004. The many faces of roc analysis in machine learning. http:\/\/www.cs.bris.ac.uk\/~flach\/ICML04tutorial\/."},{"key":"e_1_2_1_15_1","unstructured":"Graham P. 2002. A plan for spam. http:\/\/www.paulgraham.com\/spam.html.  Graham P. 2002. A plan for spam. http:\/\/www.paulgraham.com\/spam.html."},{"key":"e_1_2_1_16_1","unstructured":"Graham P. 2004. Better Bayesian filtering. http:\/\/www.paulgraham.com\/better.html.  Graham P. 2004. Better Bayesian filtering. http:\/\/www.paulgraham.com\/better.html."},{"key":"e_1_2_1_17_1","unstructured":"Graham-Cumming J. 2006. Does Bayesian poising exist&quest; Virus Bulletin.  Graham-Cumming J. 2006. Does Bayesian poising exist&quest; Virus Bulletin."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/508791.508911"},{"key":"e_1_2_1_19_1","first-page":"68","article-title":"Spam filtering II","volume":"9","author":"Holden S.","year":"2004","journal-title":"Hackin"},{"key":"e_1_2_1_20_1","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm S.","year":"1979","journal-title":"Scand. J. Stat."},{"key":"e_1_2_1_21_1","unstructured":"Kohavi R. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In IJCAI. 1137--1145.   Kohavi R. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In IJCAI. 1137--1145."},{"key":"e_1_2_1_22_1","unstructured":"Kolcz A. and Alspector J. 2001. SVM-based filtering of e-mail spam with content-specific misclassification costs. In Proceedings of the TextDM'01 Workshop on Text Mining---held at the 2001 IEEE International Conference on Data Mining. IEEE Computer Society Press Los Alamitos CA. (Available: http:\/\/www-ai.ijs.si\/DunjaMladenic\/TextDM01\/papers\/Kolcz_TM.pdf.)  Kolcz A. and Alspector J. 2001. SVM-based filtering of e-mail spam with content-specific misclassification costs. In Proceedings of the TextDM'01 Workshop on Text Mining---held at the 2001 IEEE International Conference on Data Mining. IEEE Computer Society Press Los Alamitos CA. (Available: http:\/\/www-ai.ijs.si\/DunjaMladenic\/TextDM01\/papers\/Kolcz_TM.pdf.)"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1093\/bjps\/axi152"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/215206.215366"},{"key":"e_1_2_1_25_1","unstructured":"Lynam T. and Cormack G. 2005. TREC Spam Filter Evaluation Tool Kit. http:\/\/plg.uwaterloo.ca\/~trlynam\/spamjig.  Lynam T. and Cormack G. 2005. TREC Spam Filter Evaluation Tool Kit. http:\/\/plg.uwaterloo.ca\/~trlynam\/spamjig."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of Eurospeech '97","author":"Martin A.","year":"1895"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the First Conference on Email and Anti-Spam (CEAS). (Available: http:\/\/www.ceas.cc\/papers-2004\/136","author":"Meyer T."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the UK Workshop on Computational Intelligence","author":"Pampapathi R. M."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3348\/kjr.2004.5.1.11"},{"key":"e_1_2_1_30_1","volume-title":"Spambayes: Bayesian anti-spam classifier in python","author":"Peters T.","year":"2004"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 15th International Conference on Machine Learning. Morgan-Kaufmann","author":"Provost F."},{"key":"e_1_2_1_32_1","unstructured":"Raymond E. S. Relson D. Andree M. and Louis G. 2004. Bogofilter. http:\/\/bogofilter.sourceforge.net\/.  Raymond E. S. Relson D. Andree M. and Louis G. 2004. Bogofilter. http:\/\/bogofilter.sourceforge.net\/."},{"key":"e_1_2_1_33_1","unstructured":"Robinson G. 2003. A statistical approach to the spam problem. Linux J. 107.   Robinson G. 2003. A statistical approach to the spam problem. Linux J. 107."},{"key":"e_1_2_1_34_1","unstructured":"Robinson G. 2004. Gary Robinson's spam rants. http:\/\/radio.weblogs.com\/0101454\/categories\/spam\/.  Robinson G. 2004. Gary Robinson's spam rants. http:\/\/radio.weblogs.com\/0101454\/categories\/spam\/."},{"key":"e_1_2_1_35_1","unstructured":"Rothman K. J. and Greenland S. 1998. Modern Epidemiology. Lippinscott Williams and Wilkins.  Rothman K. J. and Greenland S. 1998. Modern Epidemiology. Lippinscott Williams and Wilkins."},{"key":"e_1_2_1_36_1","unstructured":"Sahami M. Dumais S. Heckerman D. and Horvitz E. 1998. A Bayesian approach to filtering junk E-mail. In Learning for Text Categorization: Papers from the 1998 Workshop. AAAI Technical Report WS-98-05 (Madison WI).  Sahami M. Dumais S. Heckerman D. and Horvitz E. 1998. A Bayesian approach to filtering junk E-mail. In Learning for Text Categorization: Papers from the 1998 Workshop. AAAI Technical Report WS-98-05 (Madison WI)."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of EMNLP-01","author":"Sakkis G."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"key":"e_1_2_1_39_1","unstructured":"SpamAssassin. 2004. The spamassassin public mail corpus. http:\/\/spamassassin.apache.org\/publiccorpus.   SpamAssassin. 2004. The spamassassin public mail corpus. http:\/\/spamassassin.apache.org\/publiccorpus."},{"key":"e_1_2_1_40_1","unstructured":"SpamAssassin. 2005. The apache spamassassin project. http:\/\/spamassassin.apache.org.  SpamAssassin. 2005. The apache spamassassin project. http:\/\/spamassassin.apache.org."},{"key":"e_1_2_1_41_1","unstructured":"Streiner N. 1986. PDQ Statistics. B.C. Decker Inc.  Streiner N. 1986. PDQ Statistics. B.C. Decker Inc."},{"key":"e_1_2_1_42_1","volume-title":"Tech. Rep. CS-2004-03","author":"Tuttle A.","year":"2004"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the Spam Conference. Available: http:\/\/crm114","author":"Yerazunis B.","year":"2004"},{"key":"e_1_2_1_44_1","unstructured":"Yerazunis W. S. 2004b. CRM114---the controllable regex mutilator. http:\/\/crm114.sourceforge.net\/.  Yerazunis W. S. 2004b. CRM114---the controllable regex mutilator. http:\/\/crm114.sourceforge.net\/."},{"key":"e_1_2_1_45_1","unstructured":"Zdziarski J. A. 2004. The DSPAM project. http:\/\/www.nuclearelephant.com\/projects\/dspam\/.  Zdziarski J. A. 2004. The DSPAM project. http:\/\/www.nuclearelephant.com\/projects\/dspam\/."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1039621.1039625"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1247715.1247717","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1247715.1247717","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T14:47:57Z","timestamp":1750258077000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1247715.1247717"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,7]]},"references-count":46,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2007,7]]}},"alternative-id":["10.1145\/1247715.1247717"],"URL":"https:\/\/doi.org\/10.1145\/1247715.1247717","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2007,7]]},"assertion":[{"value":"2007-07-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}