You can subscribe to this list here.
2004 |
Jan
|
Feb
|
Mar
(4) |
Apr
(3) |
May
|
Jun
(2) |
Jul
|
Aug
(1) |
Sep
(2) |
Oct
|
Nov
|
Dec
(1) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2005 |
Jan
|
Feb
|
Mar
|
Apr
(5) |
May
(1) |
Jun
|
Jul
(1) |
Aug
(1) |
Sep
|
Oct
(1) |
Nov
|
Dec
|
2006 |
Jan
|
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2007 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: <m_h...@gm...> - 2007-03-29 13:11:24
|
Hello I have just compiled crm114, there was no problem visible. A test on the commandline was ok: > X-CRM114-Version: 20070301-BlameBaltar ( TRE 0.7.5 (LGPL) ) MR-BD9991E2 > X-CRM114-CacheID: sfid-20070328_230929_066681_D47F7285 > X-CRM114-Status: UNSURE ( 0.00 ) > X-CRM114-Notice: Please train this message. But wenn I setup the filters in KMail I get all mails turned back as UNSURE. I use this to check my mails: crm -u $HOME/.crm114 mailreaver.crm To move the unsure mails to spam, if they are, I use: crm -u $HOME/.crm114 mailreaver.crm --spam To move the unsure mails to ham, if they are, I use: crm -u $HOME/.crm114 mailreaver.crm --good But when I then send the same mails though the same filters again, they are still detected as UNSURE Guess I need some help at this point. I assume that I have somewhere a problem in my setup.. Thanks a lot Michael -- "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ... Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail |
From: Bill Y <ws...@me...> - 2006-07-04 22:21:10
|
There's a new Seekrit Version on the website. This one fixes the last known bug in the runtime engine (thanks, Robert!) and cleans up all outstanding issues and even has the documentation and HOWTO cleaned up to describe mailreaver and mailtrainer. Give this one a spin; if nobody has a problem with it in a day or two I will make it Offishul on the web page as "Recommended". http://crm114.sourceforge.net/crm114-20060704-BlameRobert.src.tar.gz http://crm114.sourceforge.net/crm114-20060704-BlameRobert.i386.tar.gz Here are the checksums: 0276cbcf67298c74d95c473cf411142d crm114-20060704-BlameRobert.i386.tar.gz ea1bf28e8205681f3be9feafe0a98a18 crm114-20060704-BlameRobert.src.tar.gz - Bill Yerazunis |
From: Bill Y <ws...@me...> - 2006-06-16 14:43:35
|
There's a new seekrit version on the webpage: 20060615-ButterBeast. The difference between this and VersionOfTheBeast is that this one has bugfixes for every known bug (except the one-character-at-an-iteration memory leakage which bothers nobody). http://crm114.sourceforge.net/crm114-20060615-ButterBeast.i386.tar.gz http://crm114.sourceforge.net/crm114-20060615-ButterBeast.src.tar.gz I'll wait for a few success reports on it before putting it up as the new public "bleeding edge" version. MD5sums are: acd624c916b53b319951f0803f35d705 crm114-20060615-ButterBeast.i386.tar.gz 61b7f2f9aa6f1ca549c25c8a0a153413 crm114-20060615-ButterBeast.src.tar.gz - Bill Yerazunis |
From: Bill Y <ws...@me...> - 2006-02-09 14:37:14
|
There's a new Seekrit Version on the website: crm114-20060209-ReaverSecondBreakfast. I'll run a few more QC things on it before we put it up publicly on the website. Get it at: http://crm114.sourceforge.net/crm114-20060209-ReaverSecondBreakfast.src.tar.gz Here's the blurb: This is the transition release to the new "Reaver" format for storing and caching the learned texts. The "Reaver" format keeps all incoming mail in a maildir-like cache (but _not_ your maildir; we don't touch that, and if you don't use maildir, that's just fine). The advantage is that the incoming mail is saved exactly as it needs to be trained, and a CacheID is added to the mail that you finally see. As long as the CacheID header is intact, you can just reference this stored copy (assuming it hasn't been purged away) You say "use the cached text" with --cache for command line users as in: bash: crm mailfilter.crm --cache --learnspam < text_with_cacheid_header or (for mail-to-myself users - forward to yourself with the CacheID header intact): command my_secret_password learnspam cache and you will train *exactly* the text should be trained, with no worry as to whether something in the system added headers, deleted suspicious text, or whatever. The "Reaver" cache hardlinks your trained data into "known spam" and "known nonspam" subdirectories, so even when the cache gets purged, you don't lose training data. Note that the :thickness: parameter now is meaningful! Anything scoring less than this amount (plus or minus) is tagged as UNSURE and will be delivered along the "good" mail path with the extra header of "Please train me!" Default thickness is 10 pR units, which gives really good results for me. Another advantage of the Reaver cache is that you can use mailtrainer.crm to run advanced training (like DSTTTR) on the full contents of your "known" and "probably" directories, to get really fast accurate .css files. After testing, we've changed the defaults on mailtrainer to only do one pass of DSTTTR. Notes to MeetTheReavers users who have already used mailtrainer.crm - we've added the --reload option and the default repeat count is now 1 pass (rather than 5). Checksums: md5sum crm114-20060209-ReaverSecondBreakfast.*.tar.gz 7ff472caddd9f667e67137c565ad528d crm114-20060209-ReaverSecondBreakfast.i386.tar.gz 438c73dd16e9e5af97eadc32b010e85c crm114-20060209-ReaverSecondBreakfast.src.tar.gz Enjoy.... -Bill Yerazunis |
From: Bill Y <ws...@me...> - 2006-02-03 16:03:28
|
Here's probably the best-tested CRM114 rev yet: 20060118-BlameTheReavers Because there were some big changes, I ran it myself for two weeks before going public with it. http://crm114.sourceforge.net/crm114-20060118-BlameTheReavers.src.tar.gz http://crm114.sourceforge.net/crm114-20060118-BlameTheReavers.i386.tar.gz Here's the update: "This is a big new functionality release- we include mailtrainer.crm as well as changing the default mailfilter.crm from Markovian to OSB. This new mailtrainer program is fed directories of example texts (one example per file), and produces optimized satistics files matched to your particular mailfilter.cf setup (each 1 meg of example takes about a minute of CPU). It even does N-fold validation. "Default training is 5-pass DSTTTTR (a Fidelis-inspired improvement of TUNE) with a thick threshold of 5.0 pR units. Worst-offender DSTTTTR training as a (very slow) option. There are also speedups and bugfixes throughout the code. Unless you really like Markovian, now is a good time to think about saving your old .css files and switching over to the new default mailfilter.crm config that uses OSB unique microgroom. Then run mailtrainer.crm on your saved spam and good mail files, and see how your accuracy jumps. I'm seeing about a four-fold increase in accuracy on the TREC SA corpus; this is hot stuff indeed. "HOWEVER, the downside is that mailtrainer.crm expects to see your spam and good training data files in a maildir-like format (one dir for spam, the other for good); this isn't directly supported by mailfilter.crm yet, so unless your mailer supports maildirs, you will need to write a little script to build your training data directories. MD5 checksums: 825f8c83cd1a5a83d4ae14db7c163368 crm114-20060118-BlameTheReavers.i386.tar.gz 2ce8d8483c844d51d45f127895bcc89f crm114-20060118-BlameTheReavers.src.tar.gz -Bill Yerazunis |
From: wsy <ws...@me...> - 2005-10-03 15:19:00
|
New version of CRM114 is now up on the web page. Here's the blurb: This is a new-feature release - the TRANSLATE statement allows tr()-like transformations of strings, but with more power, like downward ranges, repeated ranges, deletion, uniquing, and unique-before-transform. This should be very handy for people working on microspams. Raul Miller wrote the basis code, so he gets the blame here. There are a few bugfixes too. The next version will be concentrating on microgrooming and W32 bugfixes, and (maybe with Ville's help) an autoconf version? Here's the checksums MD5 checksums: ecb8422caad4fc5d8be2550f19b50455 crm114-20051001-BlameRaulMiller.i386.tar.gz 02be78ac900e4d78db64dcc275348c00 crm114-20051001-BlameRaulMiller.src.tar.gz -Bill Yerazunis - --j93FCcGK028670.1128352358/cambridge.merl.com-- ------- End of forwarded message ------- |
From: wsy <ws...@me...> - 2005-08-01 13:14:01
|
We have an information request on CRM114 use. If you're using CRM114 as: 1) as a mail filter for an ISP (or servicing more than a couple of dozen people at a site) OR 2) as a filter for something else ( Usenet, Web proxying, etc) for more than a few people then please drop me a note. All I need to know is *roughly* how many people are using it, and the vague category of what you're filtering. (this is for Shalen's thesis) -Bill Yerazunis |
From: wsy <ws...@me...> - 2005-07-01 01:28:06
|
Since nobody has thrown down a penalty flag on it, I'm putting the new version of CRM114 up on Sourceforge. It's working fine for me. This new version has the <unigram> and <hyperspace> classifiers. It does NOT fix the cssmerge bug (sorry) but does fix the mmap bug. Other than that, only minor bug tweaks have come in on this rev. Shalen is doing an SVM but that didn't make this release. Grab it before the rush: http://crm114.sourceforge.net/crm114-20050628.BlameCochrane.i386.tar.gz http://crm114.sourceforge.net/crm114-20050628.BlameCochrane.src.tar.gz Here's the blurb: This release debuts two new classification options - UNIGRAM and HYPERSPACE. It also contains some bugfixes (including hopefully a bugfix for the mmap error-catching problem) Using <unigram> classification effectively turns CRM114 into a normal Bayesian classifier, as <unigram> tells the classifers to use only unigrams (single words) as features. This is so people can do an apples-to-apples comparison of CRM114's Markovian, OSB Winnow, and OSB Hyperspace classifiers versus Typical Bayesian classification and not have to write tons of glue code; just change your classify/learn flag. The other (and maybe bigger) news is the new hyperspatial classifier. The hyperspatial classifier is most easily explained by analogy to astronomy. Each known-type example document represents a single star in a hyperspace of about four billion dimensions. Similar documents form clusters of light sources, like galaxies of stars. Each class of document is therefore represented by several galaxies of stars (each galaxy being documents that are hyperspatially very similar). The unknown document is an observer dropped somewhere into this four-billion-dimensional hyperspace; whichever set of galaxies sheds a brighter light on this observer is the class of the unknown document. What's amazing is that this hyperspace approach is not only faster than Bayesian OSB, it's also more accurate (better even than Winnow) and uses only about 1/40th the disk space of Markovian (300 Kbytes v. 12 Mbytes). It wants single-sided thick threshold training (SSTTT), with a thickness of 0.5 pR units. The only downside is that this is still an experimental design; use it for fun, not for production, as the file format will undoubtedly change in the future and if you don't keep your training set as individual disjoint documents you'll have to start again. Activate it by using <hyperspace> as the classify/learn control flag; you can also use <unique> and <unigram> if you want. checksums: fc3a42aacae10f0514808f89c5c6de1f crm114-20050628.BlameCochrane.i386.tar.gz 114f09866afa6f6e2b7f60b83d79718d crm114-20050628.BlameCochrane.src.tar.gz |
From: wsy <ws...@me...> - 2005-05-21 15:05:38
|
The new version of CRM114 is out on the website. The new version is 20050518.BlameMercury. Just get it off the website. http://crm114.sourceforge.net Here's the blurb: This version is a documentation primary release - CRM114 Revealed (.pdf.gz, 240 pages) is now available for download. BlameMercury has lots of bugfixes and only three extensions - you can now demonize minions onto pipes, you can re-execute failing commands from a TRAP finish, and you can now use a regex directly as a var-restriction subscripting action, so [ :my_var: /abc.*xyz/ ] gets you the matching substring in :my_var: . Var-restriction matches do NOT change the "previous MATCH" data on a variable. You should *especially* get the new version and the new documentation if you are anything beyond an "appliance user". The book is pretty much ready; it's still "proofreader version" and lacks the index, but other than that, it's *all* there. It even builds a spam filter from scratch, in 66 lines of CRM114. -Bill Yerazunis |
From: wsy <ws...@me...> - 2005-04-20 17:31:16
|
From: Fidelis Assis <fi...@po...> wsy wrote: > New release, just up on Sourceforge. > > This is a public release. > > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz [...] > > A bug in the microgroomer that causes problems when both > microgrooming and unique were used in the same learning scenario was > squashed in Markovian, OSB, and Winnow learners (it remains in OSBF). That problem had already been fixed in OSBF (patch for SanAndreas). It remained in Winnow until last seekrit. OK. I fixed Winnow. So it's fixed now. :) Good thing I took it out of the Freshmeat announcement in anticipation of your speedy fix. :) -Bill Yerazunis -- Fidelis Assis |
From: Fidelis A. <fi...@po...> - 2005-04-20 16:36:14
|
wsy wrote: > New release, just up on Sourceforge. > > This is a public release. > > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz [...] > > A bug in the microgroomer that causes problems when both > microgrooming and unique were used in the same learning scenario was > squashed in Markovian, OSB, and Winnow learners (it remains in OSBF). That problem had already been fixed in OSBF (patch for SanAndreas). It remained in Winnow until last seekrit. -- Fidelis Assis |
From: wsy <ws...@me...> - 2005-04-20 13:05:35
|
New release, just up on Sourceforge. This is a public release. http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz Here's the blurb: Math expressions can now set whether they are algebraic or RPN by a leading A or N as the first character of the string. Listings are now controllable with -l N, from simple prettyprinting to full JIT parse. A bug in the microgroomer that causes problems when both microgrooming and unique were used in the same learning scenario was squashed in Markovian, OSB, and Winnow learners (it remains in OSBF). Dependency on formail (part of procmail) in the default mailfilter.crm has been removed. A cleaner method of call-by-name, the :+: indirection-fetch operator, has been activated. The var :_cd: give the call depth for non tail recursive routines. Minor bugs have been fixed and minor speedups added. "make uninstall" works. Documentation of regexes has been improved. Cached .css mapping activated. Win32 mmap adapters inserted. I'll update the web pages ASAP. MD5 checksums: baab871b7a6bdee685e5ac6491fec247 crm114-20050415.BlameTheIRS.i386.tar.gz af42f9de1d9de55edd48d3e7f57df6fb crm114-20050415.BlameTheIRS.src.tar.gz -Bill Yerazunis |
From: wsy <ws...@me...> - 2005-04-14 23:44:30
|
New seekrit edition of CRM114 on Sourceforge: http://crm114.sourceforge.net/crm114-20050413.BlameJaspan.src.tar.gz http://crm114.sourceforge.net/crm114-20050413.BlameJaspan.i386.tar.gz Here's the blurb. Note LOTS of cool stuff made it in, including Barry's .css cacheing and Win32 adapters (which I modified but could not test). Math expressions can now set whether they are algebraic or RPN by a leading A or N as the first character of the string. Listings are now controllable with -l N, from simple prettyprinting to full JIT parse. A bug in the microgroomer that causes problems when both microgrooming and unique were used in the same learning scenario was squashed in both Markovian and OSB learners (it remains in Winnow). Dependency on formail (part of procmail) in the default mailfilter.crm has been removed. A cleaner method of call-by-name, the :+: indirection-fetch operator, has been activated. The var :_cd: give the call depth for non tail recursive routines. Minor bugs have been fixed and minor speedups added. "make uninstall" works. Documentation of regexes has been improved. Cached .css mapping activated. Win32 adapters inserted. Here are the checksums: a2b30f04ea7915c1afea3e662c5edd62 crm114-20050413.BlameJaspan.i386.tar.gz 869ee323348c1bcbd2e60541a96b010d crm114-20050413.BlameJaspan.src.tar.gz Thanks.... -Bill Yerazunis |
From: wsy <ws...@me...> - 2005-04-05 01:23:14
|
New seekrit version up on the web site - as usual, this is a "taste the dog food" release. Grab it at: http://crm114.sourceforge.net/crm114-20050402.BlameBigler.src.tar.gz http://crm114.sourceforge.net/crm114-20050402.BlameBigler.i386.tar.gz Here's the update list: Version 20050402.BlameBigler (TRE 0.7.2) Math expressions can now set whether they are algebraic or RPN by a leading A or N as the first character of the string. Listings are now controllable with -l N, from simple prettyprinting to full JIT parse. A bug in the microgroomer that causes problems when both microgrooming and unique were used in the same learning scenario was squashed in both Markovian and OSB learners. Dependency on formail (part of procmail) in the default mailfilter.crm has been removed. A cleaner method of call-by-name, the :+: indirection-fetch operator, has been activated. Minor bugs have been fixed and minor speedups added. "make uninstall" works. Documentation of regexes has been improved. The MD5 checksums are: 0d2dfc22e92c939dce40c02dfd6ef998 crm114-20050402.BlameBigler.i386.tar.gz ef40c7769fda381c51202775ed6d7929 crm114-20050402.BlameBigler.src.tar.gz -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-12-09 11:30:58
|
New release on the website: 20041209.BlameFidelisEvenMore in both .src and .i386 flavors This is the new "do not put fingers in the meat grinder" version, but it's pretty darn solid. Here's the long-form blurb: Major topics: New highest-accuracy voodoo OSBF classifier (from Fidelis Assis), Most of the new OSBF upgrades have been rolled back into OSB, including Fidelis' microgroom algorithm and CF factor, so Monkeyplexer can use these in N-way classifications; CALL/RETURN now work less unintuitively (and have actual calling _arguments_ now); SYSCALL can now fork the local process with very low overhead (excellent for demons that spawn a new instance for each connection), WINDOW has BYCHUNK for really fast block IO (by request- who wanted this again?), The FEATURECOUNT bug is fixed Details: This new revision has Fidelis Assis' new OSBF local confidence factor generator; with the OSB front end and single-sided threshold training with pR of roughly 10, it is more than 3 times more accurate and 6 times faster than straight SBPH Markovian and uses 1/10th the file space. The only downsides are that the OSBF file format is incompatible and not interconvertable between .css files and OSBF .cfc files, and that you _must_ use single-sided threshold training to achieve this accuracy. Single-sided threshold training means that if a particular text didn't score above a certain pR value, it gets trained even if it was classified correctly. For the current formulation of OSBF, training all nonspams with pR's less than 10, and all spams with pR's greater than -10 yields a very impressive 17 errors on the SA torture test, versus 42 errors with Winnow (doublesided threshold with a threshold of 0.5) and straight Markovian (54 errors with Train Only Errors training, equivalent to singlesided training with a threshold of zero) We also have several improvement in non-voodoo OSB, which gets down to 22 errors on the same torture test, again with the same training regimen (train if you aren't at least 10 pR units "sure", and _is_ upward compatible with prior OSB and Markovian .css files, and with the same speed as OSBF. It also doesn't use the "voodoo exponential confidence factor", so it may be a more general solution (on parsimony grounds); it has similar properties to OSB. The featurecount = 1 bug is fixed. CLASSIFY and LEARN both default to the obvious tokenize regex of /[[:graph:]]+/. CALL now takes three parameters: CALL /:routine_to_call:/ [:downcall_concat:] (:return_var:) The routine itself gets one parameter, which is the concatenation of all downcall_concat args (use a MATCH to shred it any way you want). RETURN now has one parameter: RETURN /:return_concat:/ The values in the :return_concat: are concatenated, and returned to the CALL statement as the new value of :return_var: ; they replace whatever was in :return_var: . SYSCALL can now fork the local process and just keep running; this saves new process invocation time, setup time, and time to run the first pass of the microcompiler. See the examples in call_return_test.crm for these new CALL / FORK / RETURN hoop-jumping tricks. WINDOW now has a new capability flag - BYCHUNK mode, specifically for users WINDOWing through large blocks of data. BYCHUNK reads as large a block of incoming data in as is available (modulo limits of the available buffer space), then applies the regex. checksums: 6225f172b19d41f70b00e9723330447f crm114-20041209.BlameFidelisEvenMore.i386.tar.gz 3d0d9775ac3f09f917b189edc436a2a1 crm114-20041209.BlameFidelisEvenMore.src.tar.gz -Bill Yerazunis |
From: Fidelis A. <fi...@po...> - 2004-09-24 19:25:54
|
Bill Yerazunis wrote: > New version on the website: > > http://crm114.sourceforge.net/20040924.BlameResidentWeasel.src.tar.gz > http://crm114.sourceforge.net/20040924.BlameResidentWeasel.i386.tar.gz Broken URL: http://crm114.sourceforge.net/crm114-20040924.BlameResidentWeasel.src.tar.gz ^^^^^^^ http://crm114.sourceforge.net/crm114-20040924.BlameResidentWeasel.i386.tar.gz ^^^^^^^ Also, that old bug is still there... // // if an integer value, print with 0 precision // if (((long)(valstack[sp]*2.0) / 2) == valstack[sp]) { - sprintf (buf, "%.0F", valstack[sp]); + sprintf (buf, "%.0f", valstack[sp]); goto formatdone; } // // if none of the above, print with three digits precision // - sprintf (buf, "%.5F", valstack[sp]); + sprintf (buf, "%.5f", valstack[sp]); // // one way or another, once we're here, we've sprinted it. -- Fidelis Assis |
From: Bill Y. <ws...@me...> - 2004-09-24 19:05:19
|
New version on the website: http://crm114.sourceforge.net/20040924.BlameResidentWeasel.src.tar.gz http://crm114.sourceforge.net/20040924.BlameResidentWeasel.i386.tar.gz The notes: Bugs stomped: several in the exit routine code, as well as fixes for detecting minion creation failures. SYSCALL can now do forking of the currently executing code; CALL is now implemented and provides CORBA-like argument transfer. A missing INSERT file in a .crm program is now a trappable error. Can use TRE version 0.7.0 (included!) Yes indeed, you can not only SYSCALL a label (which is a fork()- like thing, you get an isolated copy of the mother-ship's data in the minion and pipe connections), but you can also just CALL (which doesn't mask local variables) and can be _REALLY fast_, as in over 3000 calls/second. And you can TRAP a missing INSERT-file, so insert files can now be optional or required, at the programmer's whim. And TRE version 0.7.0 has \Q, \E, and (?inr-inr:regex) modifications, very handy for people fleeing PERL. :) Checksums: f57a20bf28c917c77ba1959b547933ca crm114-20040924.BlameResidentWeasel.i386.tar.gz 061ffbca3e34d39d96b7b1ce37981385 crm114-20040924.BlameResidentWeasel.src.tar.gz -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-08-09 00:59:21
|
New release - crm114-20040808.BlamePekingAcrobats - it's on the website, and not a seekrit release. Here's the synopsys: This is a bugfix/performance improvement release. The bugs are minor edge cases, but better _is_ better. SYSCALL now has better code for "async" and "keep" (async is now truly "fire and forget"; keep keeps the process around without losing state, and default processes will now not hang forever if they overrun the buffer.) Documentation has been improved. Both OSB and Winnow are now both faster and more accurate (as bugs were removed). A particularly nasty bug that mashed isolated vars of zero length was quashed. -D and -R (Dump and Restore) are available in cssutil for moving .css files between different-endian architectures. ----- One very nice commentary - at least half of the good things in this release are from people sending me patches! Thanks to you all! ----- Grab it off the website (this is not a seekrit release). http://crm114.sourceforge.net/ -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-06-26 01:01:58
|
New sekrit version on the website. http://crm114.sourceforge.net/crm114-20040625-BlameSiefkes MD5sum: 692c410ddb40b525bafb05b91ee4ba2d crm114-20040625-BlameSeifkes.src.tar.gz OK, I think I've gotten most of the bugs out of OSB-Winnow (thanks, Christian Siefkes, for coming up with the OSB-Winnow abomination, which earned you the blame for this release). The stats are pretty durn good. Using the standard test set (the SpamAssassin one) SBPH-Markovian gets 54 errors in TUNE, 56 in TOE OSB-Winnow gets 41 errors in TOE. (promote 1.23, demote 0.83, thickness 0.05) It seems to be winning. :) Interesting side bit- somehow I'm tickling a bug in normalizemime, as it greatly _increases_ errors when I use it. Maybe I've got a bad distro of it, because when I feed a mime'd mail into it, it does NOT undo the base64 contents, even when it's just plain text inside the base64. Jakko, should it do that? This is version 2004-02-04. --------------------------------------------------- Warning- OSB-Winnow data files are NOT COMPATIBLE with Markovian files, and cssutil/cssdiff/cssmerge does NOT work on them, although they think they do! This is all very experimental. But I'm going to put it up anyway so people can beat upon it. If you want to use it, you should also learn about "thickness learning", where you learn into one file or the other proportionally based on an error criterion. More on that when I figure it out better and can put it into a script that anyone can use. -Bill Yerazunis ---------------------------------------- Here's the notes: Besides the usual minor bugfixes (thanks!) there are two big new features in this revision: 1) We now test against ( and ship with ) TRE version 0.6.8 . Better, faster, all that. :) 2) A fourth new classifier with very impressive statistics is now available. This is the OSB-Winnow classifier, originally designed by Christian Siefkes. It combines the OSB frontend with a balanced Winnow backend. But it may well be twice as accurate as SBPH Markovian and four times more accurate than Bayesian. Like correlative matching, it does NOT produce a direct probability, but it does produce a pR, and it's integrated into the CLASSIFY statement. You invoke it with the <winnow> flag: classify <winnow> (file1.owf | file2.owf) /token_regex/ and learn <winnow> (file1.owf) /token_regex/ learn <winnow refute> (file2.owf) /token_regex/ Note that you MUST do two learns on a Winnow .owf files- one "positive" one on the correct class, and a "refute" learn on the incorrect class (actually, it's more complicated than that and I'm still working out the details and optimal settings.) Being experimental, the OSB-Winnow file format is NOT compatible with Markovian, OSB, nor correlator matching, and there's no functional checking mechanism to verify you haven't mixed up a .owf file with a .css file. Cssutil, cssdiff, and cssmerge think they can handle the new format- but they can't. Further, you currently have to train it in a two-step process, learning it into one file, and refuting it in all other files: LEARN <winnow> (file1.owf) /regex/ then LEARN <winnow refute> (file2.owf) /regex/ which will do the right thing. If the OSB-winnow system works as well as we hope, we may put the work into adding CLASSIFY-like multifile syntax into the LEARN statement so you don't have to do this two-step dance. |
From: Bill Y. <ws...@me...> - 2004-06-01 20:20:46
|
There's a new new release of CRM114 on the website. It's "bleeding edge" (i.e. public). This is release 20040601-BlameKyoto Here's the README text: This is the new bleeding edge release. It has several submitted bugfixes (attachments, windowing, wedging on microgroom), major speedups in data I/O, two new classifiers ( osb and correlate ) which can be very fast (OSB is ~2x faster than Markovian), or powerful ( correlate is a very VERY powerful classifier, and in fact if you don't have "out of band" data, it may in fact be the most accurate classifier possible; (it works on any language, any character width or encoding, on binaries, and on virii), but it's 100x - 1000x slower than Markovian). The language now allows random-access file I/O (detailed syntax can be found on the QUICKREF text). For example, if you wanted to read 16 bytes starting at byte 32 of an MP3 file (to grab one of the ID3 tags), you could say input [myfile.mp3 32 16] (:some_tag:) Likewise, you can specify an fseek and count on output as well; to overwrite the above ID3 tag, use: output [myfile.mp3 32 16] /My New Tag Text/ As usual for a bleeding-edge release, this code -is- poorly tested yet. Caution is advised. There's still a known memory leak if you reassign (via MATCH) a variable that was isolated; the short-term fix is to MATCH with another var and then ALTER the isolated copy. MD5 checksums for the tinfoil-coiffed: d8d17d75ce57da1145560279bcc6c8d5 crm114-20040601-BlameKyoto.i386.tar.gz 3dff1601f8bf2c42e8c767900fca2a56 crm114-20040601-BlameKyoto.src.tar.gz -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-04-18 13:38:50
|
There's a new bleeding-edge version up on the website. Here's the blurb: This is the new bleeding edge release. It has several submitted bugfixes (attachments, windowing), major speedups in data I/O, and now allows random-access file I/O (detailed syntax can be found on the QUICKREF text). For example, if you wanted to read 16 bytes starting at byte 32 of an MP3 file (to grab one of the ID3 tags), you could say input [myfile.mp3 32 16] (:some_tag:) Likewise, you can specify an fseek and count on output as well; to overwrite the above ID3 tag, use: output [myfile.mp3 32 16] /My New Tag Text/ This random-access file I/O means that we do NOT EVER need to add the concept of an open file to CRM114. We open, seek, read/write, and close as atomically as possible within the bounds of the file system API. I like that. :) As usual for a bleeding-edge release, this code -is- poorly tested yet. Caution is advised. There's still a known memory leak if you reassign (via MATCH) a variable that was isolated; the short-term fix is to MATCH with another var and then ALTER the isolated copy. checksums for the tinfoil-coiffed: 54337c35738d7eee8d2930b1c2a015dd crm114-20040418-BlameEasterBunny.i386.tar.gz 8d2174969d9f8a82788ee2bc36d75044 crm114-20040418-BlameEasterBunny.src.tar.gz Download it at: http://crm114.sourceforge.net/crm114-20040418-BlameEasterBunny.src.tar.gz http://crm114.sourceforge.net/crm114-20040418-BlameEasterBunny.i386.tar.gz Additionally, BlameStPatrick is now "Cutting Edge". (I'll add that even though this is "bleeding edge", it's probably been better tested than StPatrick, as I used it for the CEAS submission data sampling and so it went through about 995 kilo-messages in the process of gathering statistics. Actually, it went over a million messages, but those were my mistakes (I told it to do things that I did not mean to tell it to do).) -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-04-09 17:29:54
|
Because of the annoying bug in 20040407-BlameKatoSan, a new Seekrit release is now up on Sourceforge. The bugs fixed are a segfault when appending to a non-existent file (the file now is created), attachment forwarding, and a glitch in version reporting. As this is a seekrit release, it's visible on the .html; you have to fetch it by knowing where to look. http://crm114.sourceforge.net/crm114-20040409-BlameMarys.src.tar.gz http://crm114.sourceforge.net/crm114-20040409-BlameMarys.i386.tar.gz http://crm114.sourceforge.net/crm114-20040409-BlameMarys.css.tar.gz Again, I would appreciate it if a couple of people could give this a try and report back, I'll put it on the webpage when it gets a few passing grades. -Bill Yerazunis Tinfoil md5 checksums: 1b0792123e2ad388f64160fec21cc2c0 crm114-20040409-BlameMarys.css.tar.gz 07d26c3fcad69d849e3db59cf041e08b crm114-20040409-BlameMarys.i386.tar.gz 55009e02e7b3c1576112999da5674ed5 crm114-20040409-BlameMarys.src.tar.gz |
From: Bill Y. <ws...@me...> - 2004-04-07 20:30:58
|
There's a new Seekrit Release of CRM114 on the webpage. TOTALLY Untested. But it's got a lot of cool bugfixes and speedups. The .gzballs are: http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.src.tar.gz http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.i386.tar.gz http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.css.tar.gz As usual, I'll wait for some other folks to confirm that it compiled for them before rotating the download pointers on Sourceforge and making this the new Bleeding Edge. Till then, it's a Seekrit Release. Changes: Better splitting, faster I/O, random-access I/O, WINDOW fixes, lots of patches (including to mailfilter, up to and including the attachment fix which I hope someone else can test). The memory leak on MATCH/ISOLATE/MATCH of the same variable is still not fixed; use the workaround of MATCHing with one var and always ALTERing another until that bug is fixed. Here are the checksums: d21ae40bf1f354ce1dfeeadf8d2f1687 crm114-20040407-BlameKatoSan.css.tar.gz f97b700c18207c1d18c3e74f8caacb13 crm114-20040407-BlameKatoSan.i386.tar.gz a8bdd181e3ff4c9f832df544157db7f2 crm114-20040407-BlameKatoSan.src.tar.gz -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-03-28 12:17:22
|
Scott points out: seems like crm_expr_isolate.c is missing. Thanks! (hoisted by my own testing procedure, I am) New version with that fix and the colon fix is now on the website as "crm114-20040328-BlameStPatrick.src.tar.gz". This .gzball also includes the colon fix for subject munging. Please forget that -20040327 ever existed. -Bill Yerazunis |
From: Bill Y. <ws...@me...> - 2004-03-27 17:17:24
|
New version's up. It's bleeding edge, but fixes the WINDOW problems and a few others. It's on the web page: http://crm114.sourceforge.net Here's the update: This is the new bleeding edge release. A complete rewrite of the WINDOW code has been done (byline and eofends are gone, eofretry and eofaccepts are in), we're integrating with TRE 0.6.6 now, and a bunch of bugs have been stomped. For those poor victims who have mailreader pipelines that alter headers, you can now put "--force" on the BASH line or "force" on the mailer command line, e.g. you can now say command mysecretpassword spam force to force learning when CRM114 thinks it doesn't need to learn. However, this code -is- poorly tested yet. Caution is advised. There's still a known memory leak if you reassign (via MATCH) a variable that was isolated; the short-term fix is to MATCH with another var and then ALTER the isolated copy. The md5sums: a023a3217b663a0badecf8899b08a427 crm114-20040327-BlameStPatrick.css.tar.gz cff5bb6fd23ae7e3fd90f8ab31fbb08f crm114-20040327-BlameStPatrick.i386.tar.gz dfaaf4d9a856b585c32d1df04006cb18 crm114-20040327-BlameStPatrick.src.tar.gz -Bill Yerazunis |