Training accuracy for CRM114

I have often blogged about the efficiency of my Spam filtering setup, I've claimed that the combined CRM114 and Spamassasin setup I have is 99.92% (99.98% when both my classifiers are sure), but I have had very little data to back that up. I still have not gotten around to automating my setup's overall accuracy, but now I have some figures on one of the two classifies in my system. Here is the data from CRM114.

First, some context: when training CRM114 using the mailtrainer command, one can specify to leave out a certain percentage of the training set in the learn phase, and run a second pass over the mails so skipped to test the accuracy of the training. The way you do this is by specifying a regular expression to match the file names. Since my training set has message numbers, it was simple to use the least significant two digits as a regexp; but I did not like the idea of always leaving out the same messages. So I now generate four sets of numbers for every training run, and can optionally reserve 0%, 1%, 2%, 4%, or 10% of all mails for the accuracy run. Usually I train with 0% reserved. When the css files are new and still changing a lot, I measure accuracy with 2% reserved, and later on, I reserve 10% in the accuracy test runs.

An interesting thing to note is the assymetry in the accuracy: CRM114 has never identified a Spam message incorrectly. This is because the training mechanism is skewed towards letting a few spam mesages slip through, rather than let a good message slip into the spam folder. I like that. So, here are the accuracy numbers for CRM114; adding in Spamassassin into the mix only improves the numbers. Also, I have always felt that a freshly learned css file is somewhat brittle -- in the sense that if one trains an unsure message, and then tried to TUNE (Train Until No Errors) the css file, a large number of runs through the training set are needed until the thing stabilizes. So it is as if the learning done initially was minimalistic, and adding the information for the new unsure message required all kinds of tweaking. After a while TOEing (Training on Errors) and TUNEing, this brittleness seems to get hammered out of the CSS files. I also expect to see accuracy rise as the css files get less brittle -- The table below starts with data from a newly minted .css file, and as you can see, the accuracy climbs, especially after I swithc to reserving 10% of mails for the accuracy run.