Menu
▾
▴
crm114-announce — Announcements of new CRM114 releases (low volume)
You can subscribe to this list here.
| 2004 |
Jan
|
Feb
|
Mar
(4) |
Apr
(3) |
May
|
Jun
(2) |
Jul
|
Aug
(1) |
Sep
(2) |
Oct
|
Nov
|
Dec
(1) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2005 |
Jan
|
Feb
|
Mar
|
Apr
(5) |
May
(1) |
Jun
|
Jul
(1) |
Aug
(1) |
Sep
|
Oct
(1) |
Nov
|
Dec
|
| 2006 |
Jan
|
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2007 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <m_h...@gm...> - 2007-03-29 13:11:24
|
Hello I have just compiled crm114, there was no problem visible. A test on the commandline was ok: > X-CRM114-Version: 20070301-BlameBaltar ( TRE 0.7.5 (LGPL) ) MR-BD9991E2 > X-CRM114-CacheID: sfid-20070328_230929_066681_D47F7285 > X-CRM114-Status: UNSURE ( 0.00 ) > X-CRM114-Notice: Please train this message. But wenn I setup the filters in KMail I get all mails turned back as UNSURE. I use this to check my mails: crm -u $HOME/.crm114 mailreaver.crm To move the unsure mails to spam, if they are, I use: crm -u $HOME/.crm114 mailreaver.crm --spam To move the unsure mails to ham, if they are, I use: crm -u $HOME/.crm114 mailreaver.crm --good But when I then send the same mails though the same filters again, they are still detected as UNSURE Guess I need some help at this point. I assume that I have somewhere a problem in my setup.. Thanks a lot Michael -- "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ... Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail |
|
From: Bill Y <ws...@me...> - 2006-07-04 22:21:10
|
There's a new Seekrit Version on the website. This one fixes the last known bug in the runtime engine (thanks, Robert!) and cleans up all outstanding issues and even has the documentation and HOWTO cleaned up to describe mailreaver and mailtrainer. Give this one a spin; if nobody has a problem with it in a day or two I will make it Offishul on the web page as "Recommended". http://crm114.sourceforge.net/crm114-20060704-BlameRobert.src.tar.gz http://crm114.sourceforge.net/crm114-20060704-BlameRobert.i386.tar.gz Here are the checksums: 0276cbcf67298c74d95c473cf411142d crm114-20060704-BlameRobert.i386.tar.gz ea1bf28e8205681f3be9feafe0a98a18 crm114-20060704-BlameRobert.src.tar.gz - Bill Yerazunis |
|
From: Bill Y <ws...@me...> - 2006-06-16 14:43:35
|
There's a new seekrit version on the webpage: 20060615-ButterBeast. The difference between this and VersionOfTheBeast is that this one has bugfixes for every known bug (except the one-character-at-an-iteration memory leakage which bothers nobody). http://crm114.sourceforge.net/crm114-20060615-ButterBeast.i386.tar.gz http://crm114.sourceforge.net/crm114-20060615-ButterBeast.src.tar.gz I'll wait for a few success reports on it before putting it up as the new public "bleeding edge" version. MD5sums are: acd624c916b53b319951f0803f35d705 crm114-20060615-ButterBeast.i386.tar.gz 61b7f2f9aa6f1ca549c25c8a0a153413 crm114-20060615-ButterBeast.src.tar.gz - Bill Yerazunis |
|
From: Bill Y <ws...@me...> - 2006-02-09 14:37:14
|
There's a new Seekrit Version on the website: crm114-20060209-ReaverSecondBreakfast. I'll run a few more QC things on it before we put it up publicly on the website. Get it at: http://crm114.sourceforge.net/crm114-20060209-ReaverSecondBreakfast.src.tar.gz Here's the blurb: This is the transition release to the new "Reaver" format for storing and caching the learned texts. The "Reaver" format keeps all incoming mail in a maildir-like cache (but _not_ your maildir; we don't touch that, and if you don't use maildir, that's just fine). The advantage is that the incoming mail is saved exactly as it needs to be trained, and a CacheID is added to the mail that you finally see. As long as the CacheID header is intact, you can just reference this stored copy (assuming it hasn't been purged away) You say "use the cached text" with --cache for command line users as in: bash: crm mailfilter.crm --cache --learnspam < text_with_cacheid_header or (for mail-to-myself users - forward to yourself with the CacheID header intact): command my_secret_password learnspam cache and you will train *exactly* the text should be trained, with no worry as to whether something in the system added headers, deleted suspicious text, or whatever. The "Reaver" cache hardlinks your trained data into "known spam" and "known nonspam" subdirectories, so even when the cache gets purged, you don't lose training data. Note that the :thickness: parameter now is meaningful! Anything scoring less than this amount (plus or minus) is tagged as UNSURE and will be delivered along the "good" mail path with the extra header of "Please train me!" Default thickness is 10 pR units, which gives really good results for me. Another advantage of the Reaver cache is that you can use mailtrainer.crm to run advanced training (like DSTTTR) on the full contents of your "known" and "probably" directories, to get really fast accurate .css files. After testing, we've changed the defaults on mailtrainer to only do one pass of DSTTTR. Notes to MeetTheReavers users who have already used mailtrainer.crm - we've added the --reload option and the default repeat count is now 1 pass (rather than 5). Checksums: md5sum crm114-20060209-ReaverSecondBreakfast.*.tar.gz 7ff472caddd9f667e67137c565ad528d crm114-20060209-ReaverSecondBreakfast.i386.tar.gz 438c73dd16e9e5af97eadc32b010e85c crm114-20060209-ReaverSecondBreakfast.src.tar.gz Enjoy.... -Bill Yerazunis |
|
From: Bill Y <ws...@me...> - 2006-02-03 16:03:28
|
Here's probably the best-tested CRM114 rev yet: 20060118-BlameTheReavers Because there were some big changes, I ran it myself for two weeks before going public with it. http://crm114.sourceforge.net/crm114-20060118-BlameTheReavers.src.tar.gz http://crm114.sourceforge.net/crm114-20060118-BlameTheReavers.i386.tar.gz Here's the update: "This is a big new functionality release- we include mailtrainer.crm as well as changing the default mailfilter.crm from Markovian to OSB. This new mailtrainer program is fed directories of example texts (one example per file), and produces optimized satistics files matched to your particular mailfilter.cf setup (each 1 meg of example takes about a minute of CPU). It even does N-fold validation. "Default training is 5-pass DSTTTTR (a Fidelis-inspired improvement of TUNE) with a thick threshold of 5.0 pR units. Worst-offender DSTTTTR training as a (very slow) option. There are also speedups and bugfixes throughout the code. Unless you really like Markovian, now is a good time to think about saving your old .css files and switching over to the new default mailfilter.crm config that uses OSB unique microgroom. Then run mailtrainer.crm on your saved spam and good mail files, and see how your accuracy jumps. I'm seeing about a four-fold increase in accuracy on the TREC SA corpus; this is hot stuff indeed. "HOWEVER, the downside is that mailtrainer.crm expects to see your spam and good training data files in a maildir-like format (one dir for spam, the other for good); this isn't directly supported by mailfilter.crm yet, so unless your mailer supports maildirs, you will need to write a little script to build your training data directories. MD5 checksums: 825f8c83cd1a5a83d4ae14db7c163368 crm114-20060118-BlameTheReavers.i386.tar.gz 2ce8d8483c844d51d45f127895bcc89f crm114-20060118-BlameTheReavers.src.tar.gz -Bill Yerazunis |
|
From: wsy <ws...@me...> - 2005-10-03 15:19:00
|
New version of CRM114 is now up on the web page. Here's the blurb: This is a new-feature release - the TRANSLATE statement allows tr()-like transformations of strings, but with more power, like downward ranges, repeated ranges, deletion, uniquing, and unique-before-transform. This should be very handy for people working on microspams. Raul Miller wrote the basis code, so he gets the blame here. There are a few bugfixes too. The next version will be concentrating on microgrooming and W32 bugfixes, and (maybe with Ville's help) an autoconf version? Here's the checksums MD5 checksums: ecb8422caad4fc5d8be2550f19b50455 crm114-20051001-BlameRaulMiller.i386.tar.gz 02be78ac900e4d78db64dcc275348c00 crm114-20051001-BlameRaulMiller.src.tar.gz -Bill Yerazunis - --j93FCcGK028670.1128352358/cambridge.merl.com-- ------- End of forwarded message ------- |
|
From: wsy <ws...@me...> - 2005-08-01 13:14:01
|
We have an information request on CRM114 use.
If you're using CRM114 as:
1) as a mail filter for an ISP (or servicing more than a couple of
dozen people at a site)
OR
2) as a filter for something else ( Usenet, Web proxying, etc) for
more than a few people
then please drop me a note.
All I need to know is *roughly* how many people are using it, and the
vague category of what you're filtering.
(this is for Shalen's thesis)
-Bill Yerazunis
|
|
From: wsy <ws...@me...> - 2005-07-01 01:28:06
|
Since nobody has thrown down a penalty flag on it, I'm putting the
new version of CRM114 up on Sourceforge. It's working fine for me.
This new version has the <unigram> and <hyperspace> classifiers. It does
NOT fix the cssmerge bug (sorry) but does fix the mmap bug. Other than
that, only minor bug tweaks have come in on this rev. Shalen is doing
an SVM but that didn't make this release.
Grab it before the rush:
http://crm114.sourceforge.net/crm114-20050628.BlameCochrane.i386.tar.gz
http://crm114.sourceforge.net/crm114-20050628.BlameCochrane.src.tar.gz
Here's the blurb:
This release debuts two new classification options - UNIGRAM and
HYPERSPACE. It also contains some bugfixes (including hopefully a
bugfix for the mmap error-catching problem)
Using <unigram> classification effectively turns CRM114 into a normal
Bayesian classifier, as <unigram> tells the classifers to use only
unigrams (single words) as features. This is so people can do an
apples-to-apples comparison of CRM114's Markovian, OSB Winnow, and OSB
Hyperspace classifiers versus Typical Bayesian classification and not
have to write tons of glue code; just change your classify/learn flag.
The other (and maybe bigger) news is the new hyperspatial classifier.
The hyperspatial classifier is most easily explained by analogy to
astronomy. Each known-type example document represents a single star
in a hyperspace of about four billion dimensions. Similar documents
form clusters of light sources, like galaxies of stars. Each class of
document is therefore represented by several galaxies of stars (each
galaxy being documents that are hyperspatially very similar). The
unknown document is an observer dropped somewhere into this
four-billion-dimensional hyperspace; whichever set of galaxies sheds a
brighter light on this observer is the class of the unknown document.
What's amazing is that this hyperspace approach is not only faster
than Bayesian OSB, it's also more accurate (better even than Winnow)
and uses only about 1/40th the disk space of Markovian (300 Kbytes
v. 12 Mbytes). It wants single-sided thick threshold training (SSTTT),
with a thickness of 0.5 pR units. The only downside is that this is
still an experimental design; use it for fun, not for production, as
the file format will undoubtedly change in the future and if you don't
keep your training set as individual disjoint documents you'll have to
start again. Activate it by using <hyperspace> as the classify/learn
control flag; you can also use <unique> and <unigram> if you want.
checksums:
fc3a42aacae10f0514808f89c5c6de1f crm114-20050628.BlameCochrane.i386.tar.gz
114f09866afa6f6e2b7f60b83d79718d crm114-20050628.BlameCochrane.src.tar.gz
|
|
From: wsy <ws...@me...> - 2005-05-21 15:05:38
|
The new version of CRM114 is out on the website. The new
version is 20050518.BlameMercury. Just get it off the website.
http://crm114.sourceforge.net
Here's the blurb:
This version is a documentation primary release - CRM114 Revealed
(.pdf.gz, 240 pages) is now available for download. BlameMercury
has lots of bugfixes and only three extensions - you can now
demonize minions onto pipes, you can re-execute failing commands
from a TRAP finish, and you can now use a regex directly as a
var-restriction subscripting action, so [ :my_var: /abc.*xyz/ ] gets
you the matching substring in :my_var: . Var-restriction matches do
NOT change the "previous MATCH" data on a variable.
You should *especially* get the new version and the new documentation
if you are anything beyond an "appliance user".
The book is pretty much ready; it's still "proofreader version" and
lacks the index, but other than that, it's *all* there.
It even builds a spam filter from scratch, in 66 lines of CRM114.
-Bill Yerazunis
|
|
From: wsy <ws...@me...> - 2005-04-20 17:31:16
|
From: Fidelis Assis <fi...@po...> wsy wrote: > New release, just up on Sourceforge. > > This is a public release. > > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz [...] > > A bug in the microgroomer that causes problems when both > microgrooming and unique were used in the same learning scenario was > squashed in Markovian, OSB, and Winnow learners (it remains in OSBF). That problem had already been fixed in OSBF (patch for SanAndreas). It remained in Winnow until last seekrit. OK. I fixed Winnow. So it's fixed now. :) Good thing I took it out of the Freshmeat announcement in anticipation of your speedy fix. :) -Bill Yerazunis -- Fidelis Assis |
|
From: Fidelis A. <fi...@po...> - 2005-04-20 16:36:14
|
wsy wrote: > New release, just up on Sourceforge. > > This is a public release. > > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz > http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz [...] > > A bug in the microgroomer that causes problems when both > microgrooming and unique were used in the same learning scenario was > squashed in Markovian, OSB, and Winnow learners (it remains in OSBF). That problem had already been fixed in OSBF (patch for SanAndreas). It remained in Winnow until last seekrit. -- Fidelis Assis |
|
From: wsy <ws...@me...> - 2005-04-20 13:05:35
|
New release, just up on Sourceforge.
This is a public release.
http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.src.tar.gz
http://crm114.sourceforge.net/crm114-20050415.BlameTheIRS.i386.tar.gz
Here's the blurb:
Math expressions can now set whether they are algebraic or RPN by a
leading A or N as the first character of the string. Listings are
now controllable with -l N, from simple prettyprinting to full JIT
parse. A bug in the microgroomer that causes problems when both
microgrooming and unique were used in the same learning scenario was
squashed in Markovian, OSB, and Winnow learners (it remains in OSBF).
Dependency on formail (part of procmail) in the default
mailfilter.crm has been removed. A cleaner method of call-by-name,
the :+: indirection-fetch operator, has been activated. The var :_cd:
give the call depth for non tail recursive routines. Minor bugs have
been fixed and minor speedups added. "make uninstall" works.
Documentation of regexes has been improved. Cached .css mapping
activated. Win32 mmap adapters inserted.
I'll update the web pages ASAP.
MD5 checksums:
baab871b7a6bdee685e5ac6491fec247 crm114-20050415.BlameTheIRS.i386.tar.gz
af42f9de1d9de55edd48d3e7f57df6fb crm114-20050415.BlameTheIRS.src.tar.gz
-Bill Yerazunis
|
|
From: wsy <ws...@me...> - 2005-04-14 23:44:30
|
New seekrit edition of CRM114 on Sourceforge:
http://crm114.sourceforge.net/crm114-20050413.BlameJaspan.src.tar.gz
http://crm114.sourceforge.net/crm114-20050413.BlameJaspan.i386.tar.gz
Here's the blurb. Note LOTS of cool stuff made it in, including Barry's
.css cacheing and Win32 adapters (which I modified but could not test).
Math expressions can now set whether they are algebraic or RPN by a
leading A or N as the first character of the string. Listings are
now controllable with -l N, from simple prettyprinting to full JIT
parse. A bug in the microgroomer that causes problems when both
microgrooming and unique were used in the same learning scenario was
squashed in both Markovian and OSB learners (it remains in Winnow).
Dependency on formail (part of procmail) in the default
mailfilter.crm has been removed. A cleaner method of call-by-name,
the :+: indirection-fetch operator, has been activated. The var
:_cd: give the call depth for non tail recursive routines. Minor
bugs have been fixed and minor speedups added. "make uninstall"
works. Documentation of regexes has been improved. Cached .css
mapping activated. Win32 adapters inserted.
Here are the checksums:
a2b30f04ea7915c1afea3e662c5edd62 crm114-20050413.BlameJaspan.i386.tar.gz
869ee323348c1bcbd2e60541a96b010d crm114-20050413.BlameJaspan.src.tar.gz
Thanks....
-Bill Yerazunis
|
|
From: wsy <ws...@me...> - 2005-04-05 01:23:14
|
New seekrit version up on the web site - as usual, this is a "taste
the dog food" release.
Grab it at:
http://crm114.sourceforge.net/crm114-20050402.BlameBigler.src.tar.gz
http://crm114.sourceforge.net/crm114-20050402.BlameBigler.i386.tar.gz
Here's the update list:
Version 20050402.BlameBigler (TRE 0.7.2)
Math expressions can now set whether they are algebraic or RPN by a
leading A or N as the first character of the string. Listings are
now controllable with -l N, from simple prettyprinting to full JIT
parse. A bug in the microgroomer that causes problems when both
microgrooming and unique were used in the same learning scenario was
squashed in both Markovian and OSB learners. Dependency on formail
(part of procmail) in the default mailfilter.crm has been removed.
A cleaner method of call-by-name, the :+: indirection-fetch
operator, has been activated. Minor bugs have been fixed and minor
speedups added. "make uninstall" works. Documentation of regexes
has been improved.
The MD5 checksums are:
0d2dfc22e92c939dce40c02dfd6ef998 crm114-20050402.BlameBigler.i386.tar.gz
ef40c7769fda381c51202775ed6d7929 crm114-20050402.BlameBigler.src.tar.gz
-Bill Yerazunis
|
|
From: Bill Y. <ws...@me...> - 2004-12-09 11:30:58
|
New release on the website:
20041209.BlameFidelisEvenMore
in both .src and .i386 flavors
This is the new "do not put fingers in the meat grinder" version, but
it's pretty darn solid.
Here's the long-form blurb:
Major topics:
New highest-accuracy voodoo OSBF classifier (from Fidelis Assis),
Most of the new OSBF upgrades have been rolled back into OSB,
including Fidelis' microgroom algorithm and CF factor, so
Monkeyplexer can use these in N-way classifications;
CALL/RETURN now work less unintuitively (and have actual calling
_arguments_ now);
SYSCALL can now fork the local process with very low overhead
(excellent for demons that spawn a new instance for each connection),
WINDOW has BYCHUNK for really fast block IO (by request- who wanted
this again?),
The FEATURECOUNT bug is fixed
Details:
This new revision has Fidelis Assis' new OSBF local confidence factor
generator; with the OSB front end and single-sided threshold training
with pR of roughly 10, it is more than 3 times more accurate and 6
times faster than straight SBPH Markovian and uses 1/10th the file
space. The only downsides are that the OSBF file format is
incompatible and not interconvertable between .css files and OSBF .cfc
files, and that you _must_ use single-sided threshold training to
achieve this accuracy. Single-sided threshold training means that if a
particular text didn't score above a certain pR value, it gets trained
even if it was classified correctly. For the current formulation of
OSBF, training all nonspams with pR's less than 10, and all spams with
pR's greater than -10 yields a very impressive 17 errors on the SA
torture test, versus 42 errors with Winnow (doublesided threshold with
a threshold of 0.5) and straight Markovian (54 errors with Train Only
Errors training, equivalent to singlesided training with a threshold
of zero)
We also have several improvement in non-voodoo OSB, which gets down to
22 errors on the same torture test, again with the same training
regimen (train if you aren't at least 10 pR units "sure", and _is_
upward compatible with prior OSB and Markovian .css files, and with
the same speed as OSBF. It also doesn't use the "voodoo exponential
confidence factor", so it may be a more general solution (on parsimony
grounds); it has similar properties to OSB. The featurecount = 1 bug
is fixed.
CLASSIFY and LEARN both default to the obvious tokenize regex of
/[[:graph:]]+/.
CALL now takes three parameters: CALL /:routine_to_call:/
[:downcall_concat:] (:return_var:) The routine itself gets one
parameter, which is the concatenation of all downcall_concat args (use
a MATCH to shred it any way you want). RETURN now has one parameter:
RETURN /:return_concat:/ The values in the :return_concat: are
concatenated, and returned to the CALL statement as the new value of
:return_var: ; they replace whatever was in :return_var: .
SYSCALL can now fork the local process and just keep running; this
saves new process invocation time, setup time, and time to run the
first pass of the microcompiler.
See the examples in call_return_test.crm for these new CALL / FORK /
RETURN hoop-jumping tricks.
WINDOW now has a new capability flag - BYCHUNK mode, specifically for
users WINDOWing through large blocks of data. BYCHUNK reads as large a
block of incoming data in as is available (modulo limits of the
available buffer space), then applies the regex.
checksums:
6225f172b19d41f70b00e9723330447f crm114-20041209.BlameFidelisEvenMore.i386.tar.gz
3d0d9775ac3f09f917b189edc436a2a1 crm114-20041209.BlameFidelisEvenMore.src.tar.gz
-Bill Yerazunis
|
|
From: Fidelis A. <fi...@po...> - 2004-09-24 19:25:54
|
Bill Yerazunis wrote: > New version on the website: > > http://crm114.sourceforge.net/20040924.BlameResidentWeasel.src.tar.gz > http://crm114.sourceforge.net/20040924.BlameResidentWeasel.i386.tar.gz Broken URL: http://crm114.sourceforge.net/crm114-20040924.BlameResidentWeasel.src.tar.gz ^^^^^^^ http://crm114.sourceforge.net/crm114-20040924.BlameResidentWeasel.i386.tar.gz ^^^^^^^ Also, that old bug is still there... // // if an integer value, print with 0 precision // if (((long)(valstack[sp]*2.0) / 2) == valstack[sp]) { - sprintf (buf, "%.0F", valstack[sp]); + sprintf (buf, "%.0f", valstack[sp]); goto formatdone; } // // if none of the above, print with three digits precision // - sprintf (buf, "%.5F", valstack[sp]); + sprintf (buf, "%.5f", valstack[sp]); // // one way or another, once we're here, we've sprinted it. -- Fidelis Assis |
|
From: Bill Y. <ws...@me...> - 2004-09-24 19:05:19
|
New version on the website: http://crm114.sourceforge.net/20040924.BlameResidentWeasel.src.tar.gz http://crm114.sourceforge.net/20040924.BlameResidentWeasel.i386.tar.gz The notes: Bugs stomped: several in the exit routine code, as well as fixes for detecting minion creation failures. SYSCALL can now do forking of the currently executing code; CALL is now implemented and provides CORBA-like argument transfer. A missing INSERT file in a .crm program is now a trappable error. Can use TRE version 0.7.0 (included!) Yes indeed, you can not only SYSCALL a label (which is a fork()- like thing, you get an isolated copy of the mother-ship's data in the minion and pipe connections), but you can also just CALL (which doesn't mask local variables) and can be _REALLY fast_, as in over 3000 calls/second. And you can TRAP a missing INSERT-file, so insert files can now be optional or required, at the programmer's whim. And TRE version 0.7.0 has \Q, \E, and (?inr-inr:regex) modifications, very handy for people fleeing PERL. :) Checksums: f57a20bf28c917c77ba1959b547933ca crm114-20040924.BlameResidentWeasel.i386.tar.gz 061ffbca3e34d39d96b7b1ce37981385 crm114-20040924.BlameResidentWeasel.src.tar.gz -Bill Yerazunis |
|
From: Bill Y. <ws...@me...> - 2004-08-09 00:59:21
|
New release - crm114-20040808.BlamePekingAcrobats - it's on the website,
and not a seekrit release.
Here's the synopsys:
This is a bugfix/performance improvement release. The bugs are
minor edge cases, but better _is_ better. SYSCALL now has better
code for "async" and "keep" (async is now truly "fire and forget";
keep keeps the process around without losing state, and default
processes will now not hang forever if they overrun the buffer.)
Documentation has been improved. Both OSB and Winnow are now both
faster and more accurate (as bugs were removed). A particularly
nasty bug that mashed isolated vars of zero length was quashed. -D
and -R (Dump and Restore) are available in cssutil for moving .css
files between different-endian architectures.
-----
One very nice commentary - at least half of the good things in
this release are from people sending me patches! Thanks to
you all!
-----
Grab it off the website (this is not a seekrit release).
http://crm114.sourceforge.net/
-Bill Yerazunis
|
|
From: Bill Y. <ws...@me...> - 2004-06-26 01:01:58
|
New sekrit version on the website.
http://crm114.sourceforge.net/crm114-20040625-BlameSiefkes
MD5sum:
692c410ddb40b525bafb05b91ee4ba2d crm114-20040625-BlameSeifkes.src.tar.gz
OK, I think I've gotten most of the bugs out of OSB-Winnow (thanks,
Christian Siefkes, for coming up with the OSB-Winnow abomination,
which earned you the blame for this release).
The stats are pretty durn good.
Using the standard test set (the SpamAssassin one)
SBPH-Markovian gets 54 errors in TUNE, 56 in TOE
OSB-Winnow gets 41 errors in TOE.
(promote 1.23, demote 0.83, thickness 0.05)
It seems to be winning. :)
Interesting side bit- somehow I'm tickling a bug in normalizemime, as
it greatly _increases_ errors when I use it. Maybe I've got a bad
distro of it, because when I feed a mime'd mail into it, it does
NOT undo the base64 contents, even when it's just plain text inside
the base64. Jakko, should it do that? This is version 2004-02-04.
---------------------------------------------------
Warning- OSB-Winnow data files are NOT COMPATIBLE with Markovian
files, and cssutil/cssdiff/cssmerge does NOT work on them, although
they think they do!
This is all very experimental. But I'm going to put it up anyway
so people can beat upon it.
If you want to use it, you should also learn about "thickness
learning", where you learn into one file or the other proportionally
based on an error criterion. More on that when I figure it out
better and can put it into a script that anyone can use.
-Bill Yerazunis
----------------------------------------
Here's the notes:
Besides the usual minor bugfixes (thanks!) there are two big new features
in this revision:
1) We now test against ( and ship with ) TRE version 0.6.8 . Better,
faster, all that. :)
2) A fourth new classifier with very impressive statistics is now
available. This is the OSB-Winnow classifier, originally designed by
Christian Siefkes. It combines the OSB frontend with a balanced
Winnow backend. But it may well be twice as accurate as SBPH Markovian
and four times more accurate than Bayesian. Like correlative matching,
it does NOT produce a direct probability, but it does produce a pR,
and it's integrated into the CLASSIFY statement. You invoke it
with the <winnow> flag:
classify <winnow> (file1.owf | file2.owf) /token_regex/
and
learn <winnow> (file1.owf) /token_regex/
learn <winnow refute> (file2.owf) /token_regex/
Note that you MUST do two learns on a Winnow .owf files- one
"positive" one on the correct class, and a "refute" learn on
the incorrect class (actually, it's more complicated than that
and I'm still working out the details and optimal settings.)
Being experimental, the OSB-Winnow file format is NOT compatible with
Markovian, OSB, nor correlator matching, and there's no functional
checking mechanism to verify you haven't mixed up a .owf file with a
.css file. Cssutil, cssdiff, and cssmerge think they can handle the
new format- but they can't.
Further, you currently have to train it in a two-step
process, learning it into one file, and refuting it in all other
files:
LEARN <winnow> (file1.owf) /regex/
then
LEARN <winnow refute> (file2.owf) /regex/
which will do the right thing. If the OSB-winnow system works as well
as we hope, we may put the work into adding CLASSIFY-like multifile
syntax into the LEARN statement so you don't have to do this two-step
dance.
|
|
From: Bill Y. <ws...@me...> - 2004-06-01 20:20:46
|
There's a new new release of CRM114 on the website. It's
"bleeding edge" (i.e. public).
This is release 20040601-BlameKyoto
Here's the README text:
This is the new bleeding edge release. It has several submitted
bugfixes (attachments, windowing, wedging on microgroom), major
speedups in data I/O, two new classifiers ( osb and correlate ) which
can be very fast (OSB is ~2x faster than Markovian), or powerful (
correlate is a very VERY powerful classifier, and in fact if you don't
have "out of band" data, it may in fact be the most accurate
classifier possible; (it works on any language, any character width or
encoding, on binaries, and on virii), but it's 100x - 1000x slower
than Markovian).
The language now allows random-access file I/O (detailed syntax can be
found on the QUICKREF text).
For example, if you wanted to read 16 bytes starting at byte 32 of an
MP3 file (to grab one of the ID3 tags), you could say
input [myfile.mp3 32 16] (:some_tag:)
Likewise, you can specify an fseek and count on output as well; to
overwrite the above ID3 tag, use:
output [myfile.mp3 32 16] /My New Tag Text/
As usual for a bleeding-edge release, this code -is- poorly tested
yet. Caution is advised. There's still a known memory leak if you
reassign (via MATCH) a variable that was isolated; the short-term fix
is to MATCH with another var and then ALTER the isolated copy.
MD5 checksums for the tinfoil-coiffed:
d8d17d75ce57da1145560279bcc6c8d5 crm114-20040601-BlameKyoto.i386.tar.gz
3dff1601f8bf2c42e8c767900fca2a56 crm114-20040601-BlameKyoto.src.tar.gz
-Bill Yerazunis
|
|
From: Bill Y. <ws...@me...> - 2004-04-18 13:38:50
|
There's a new bleeding-edge version up on the website. Here's the blurb:
This is the new bleeding edge release. It has several submitted
bugfixes (attachments, windowing), major speedups in data I/O, and
now allows random-access file I/O (detailed syntax can be found on
the QUICKREF text). For example, if you wanted to read 16 bytes
starting at byte 32 of an MP3 file (to grab one of the ID3 tags),
you could say
input [myfile.mp3 32 16] (:some_tag:)
Likewise, you can specify an fseek and count on output as well; to
overwrite the above ID3 tag, use:
output [myfile.mp3 32 16] /My New Tag Text/
This random-access file I/O means that we do NOT EVER need to add the
concept of an open file to CRM114. We open, seek, read/write, and
close as atomically as possible within the bounds of the file system
API. I like that. :)
As usual for a bleeding-edge release, this code -is- poorly tested
yet. Caution is advised. There's still a known memory leak if you
reassign (via MATCH) a variable that was isolated; the short-term
fix is to MATCH with another var and then ALTER the isolated copy.
checksums for the tinfoil-coiffed:
54337c35738d7eee8d2930b1c2a015dd crm114-20040418-BlameEasterBunny.i386.tar.gz
8d2174969d9f8a82788ee2bc36d75044 crm114-20040418-BlameEasterBunny.src.tar.gz
Download it at:
http://crm114.sourceforge.net/crm114-20040418-BlameEasterBunny.src.tar.gz
http://crm114.sourceforge.net/crm114-20040418-BlameEasterBunny.i386.tar.gz
Additionally, BlameStPatrick is now "Cutting Edge".
(I'll add that even though this is "bleeding edge", it's probably been
better tested than StPatrick, as I used it for the CEAS submission data
sampling and so it went through about 995 kilo-messages in the process
of gathering statistics. Actually, it went over a million messages, but
those were my mistakes (I told it to do things that I did not mean to tell
it to do).)
-Bill Yerazunis
|
|
From: Bill Y. <ws...@me...> - 2004-04-09 17:29:54
|
Because of the annoying bug in 20040407-BlameKatoSan, a new Seekrit release is now up on Sourceforge. The bugs fixed are a segfault when appending to a non-existent file (the file now is created), attachment forwarding, and a glitch in version reporting. As this is a seekrit release, it's visible on the .html; you have to fetch it by knowing where to look. http://crm114.sourceforge.net/crm114-20040409-BlameMarys.src.tar.gz http://crm114.sourceforge.net/crm114-20040409-BlameMarys.i386.tar.gz http://crm114.sourceforge.net/crm114-20040409-BlameMarys.css.tar.gz Again, I would appreciate it if a couple of people could give this a try and report back, I'll put it on the webpage when it gets a few passing grades. -Bill Yerazunis Tinfoil md5 checksums: 1b0792123e2ad388f64160fec21cc2c0 crm114-20040409-BlameMarys.css.tar.gz 07d26c3fcad69d849e3db59cf041e08b crm114-20040409-BlameMarys.i386.tar.gz 55009e02e7b3c1576112999da5674ed5 crm114-20040409-BlameMarys.src.tar.gz |
|
From: Bill Y. <ws...@me...> - 2004-04-07 20:30:58
|
There's a new Seekrit Release of CRM114 on the webpage.
TOTALLY Untested. But it's got a lot of cool bugfixes and speedups.
The .gzballs are:
http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.src.tar.gz
http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.i386.tar.gz
http://crm114.sourceforge.net/crm114-20040407-BlameKatoSan.css.tar.gz
As usual, I'll wait for some other folks to confirm that it compiled
for them before rotating the download pointers on Sourceforge and
making this the new Bleeding Edge. Till then, it's a Seekrit Release.
Changes: Better splitting, faster I/O, random-access I/O, WINDOW fixes,
lots of patches (including to mailfilter, up to and including the
attachment fix which I hope someone else can test).
The memory leak on MATCH/ISOLATE/MATCH of the same variable is still
not fixed; use the workaround of MATCHing with one var and always
ALTERing another until that bug is fixed.
Here are the checksums:
d21ae40bf1f354ce1dfeeadf8d2f1687 crm114-20040407-BlameKatoSan.css.tar.gz
f97b700c18207c1d18c3e74f8caacb13 crm114-20040407-BlameKatoSan.i386.tar.gz
a8bdd181e3ff4c9f832df544157db7f2 crm114-20040407-BlameKatoSan.src.tar.gz
-Bill Yerazunis
|
|
From: Bill Y. <ws...@me...> - 2004-03-28 12:17:22
|
Scott points out: seems like crm_expr_isolate.c is missing. Thanks! (hoisted by my own testing procedure, I am) New version with that fix and the colon fix is now on the website as "crm114-20040328-BlameStPatrick.src.tar.gz". This .gzball also includes the colon fix for subject munging. Please forget that -20040327 ever existed. -Bill Yerazunis |
|
From: Bill Y. <ws...@me...> - 2004-03-27 17:17:24
|
New version's up. It's bleeding edge, but fixes the WINDOW problems and
a few others.
It's on the web page:
http://crm114.sourceforge.net
Here's the update:
This is the new bleeding edge release. A complete rewrite of
the WINDOW code has been done (byline and eofends are gone, eofretry
and eofaccepts are in), we're integrating with TRE 0.6.6 now,
and a bunch of bugs have been stomped.
For those poor victims who have mailreader pipelines that alter headers,
you can now put "--force" on the BASH line or "force" on the
mailer command line, e.g. you can now say
command mysecretpassword spam force
to force learning when CRM114 thinks it doesn't need to learn.
However, this code -is- poorly tested yet. Caution is advised.
There's still a known memory leak if you reassign (via MATCH) a
variable that was isolated; the short-term fix is to MATCH with
another var and then ALTER the isolated copy.
The md5sums:
a023a3217b663a0badecf8899b08a427 crm114-20040327-BlameStPatrick.css.tar.gz
cff5bb6fd23ae7e3fd90f8ab31fbb08f crm114-20040327-BlameStPatrick.i386.tar.gz
dfaaf4d9a856b585c32d1df04006cb18 crm114-20040327-BlameStPatrick.src.tar.gz
-Bill Yerazunis
|