SpamProbe - fast bayesian spam filter

Sorry guys, but this is the smartest email group I know.

I'm having some clients who are suddenly getting emails apparently with 
malformed headers which hangs their email clients. The clients just act 
like no more mail is on the server. They can go in via Webmail and 
delete 'some mail' and then suddenly their email will download. Any ideas?

I can tell you that my SpamProbe clients are NOT suffering from this as 
I assume these emails are spam and since I'm putting spam into an IMAP 
box, the email clients seem to handle the mail fine. This seems to be 
only on POP accounts and so far I have not been able to determine where 
the mail is hanging. I run sendmail. Got a report from another person 
today that they are having the same problem and they are on Qmail. Seems 
like the email client thinks the rest of the mail is read and marked to 
be left on the server or some such. Very odd.

Sorry for the side topic. Anybody have a solution, would love a reply, 
offlist, unless others would like to know about this as well.

Best,
John Hinton

I've seen this before ... but never had a perfect answer.
My *guess* was the email box was just too big ... 
and by cutting down on the size the client started to work again.
But as you guess ... maybe malformed headers are doing the trick ?
Maybe this is the latest wave of worm/virus emails ?



> Sorry guys, but this is the smartest email group I know.
> 
> I'm having some clients who are suddenly getting emails apparently with 
> malformed headers which hangs their email clients. The clients just act 
> like no more mail is on the server. They can go in via Webmail and 
> delete 'some mail' and then suddenly their email will download. Any ideas?
> 
> I can tell you that my SpamProbe clients are NOT suffering from this as 
> I assume these emails are spam and since I'm putting spam into an IMAP 
> box, the email clients seem to handle the mail fine. This seems to be 
> only on POP accounts and so far I have not been able to determine where 
> the mail is hanging. I run sendmail. Got a report from another person 
> today that they are having the same problem and they are on Qmail. Seems 
> like the email client thinks the rest of the mail is read and marked to 
> be left on the server or some such. Very odd.
> 
> Sorry for the side topic. Anybody have a solution, would love a reply, 
> offlist, unless others would like to know about this as well.
> 
> Best,
> John Hinton
> 
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Spamprobe-users mailing list
> Spamprobe-users@...
> https://lists.sourceforge.net/lists/listinfo/spamprobe-users
>

On Tue, 27 Jan 2004, Jem wrote:
> I found spamprobe wasn't learning that these things were spam (they only
> contain an attachment so they're isn't much data to go on). I'm sharing my

Hmmm...maybe spamprobe should consider lines that match the RE
"^[a-zA-Z0-9+/]+$" to be a token?

-- 
--tzs

On Tue, 27 Jan 2004 10:19:10 -0500 "David A. Lee" <dlee@...>
wrote:

> I've seen this before ... but never had a perfect answer.
> My *guess* was the email box was just too big ... 
> and by cutting down on the size the client started to work again.
> But as you guess ... maybe malformed headers are doing the trick ?
> Maybe this is the latest wave of worm/virus emails ?

I experienced this too, once. Mailbox size was definitely not the problem.
My mail client apparently broke off the POP connection while receiving the
broken message. I think I "solved" this by deleting the message through
Webmail.

MSB

-- 
The best time to plant a tree was 20 years ago - 

The second best is now!

i get that sometimes using courier-mta as well. it SEEM to be a
malformed message (The Great Sam says extra \r though i haven't been
able to see that myself) or a big message (sometimes 1 megs will jam the
box, at some other times it's 3 megs, and sometimes 10+ megs message
will go through just fine)

perlstalker on the #courier also suggested malformed mime sections, he
wrote a small perl script to check that in perl. i can try and find it
if you want 

On Tue, 2004-01-27 at 14:41, Matthias Benkmann wrote:
> On Tue, 27 Jan 2004 10:19:10 -0500 "David A. Lee" <dlee@...>
> wrote:
> 
> > I've seen this before ... but never had a perfect answer.
> > My *guess* was the email box was just too big ... 
> > and by cutting down on the size the client started to work again.
> > But as you guess ... maybe malformed headers are doing the trick ?
> > Maybe this is the latest wave of worm/virus emails ?
> 
> I experienced this too, once. Mailbox size was definitely not the problem.
> My mail client apparently broke off the POP connection while receiving the
> broken message. I think I "solved" this by deleting the message through
> Webmail.
> 
> MSB

At 08:11 27/01/2004, Jem wrote:

>I found spamprobe wasn't learning that these things were spam (they only
>contain an attachment so they're isn't much data to go on). I'm sharing my
>procmail rules I've derived by extracting arbitrary IDs from the various
>strains I've received:
>
>http://wpbl.pc9.org/procmailrc
>
>In my rules, upon detection of the worm the message is dropped to the WPBL
>data collection tool (this records the abusive IP and reports it to our
>central server -- ends up on wpbl.dnsbl.net.au). If you're not a WPBL member
>you will probably want to replace the findip call with /dev/null
>

Wouldn't it be better just to use an virus scanner?  I'm using ClamAV 
(http://www.clamav.net/) and it works very well.  It downloaded updated 
virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.

I have it running from a Procmail rule along side SpamProbe without any 
problems.  Best of all, of course, is that it's free!

Mike


-- 
Michael Nolan
michael@...
http://www.michaelnolan.co.uk/

On Wed, 28 Jan 2004 09:54:42 +0000 Michael Nolan
<michael@...> wrote:

> Wouldn't it be better just to use an virus scanner? 

I agree. I see no way to Bayes filter false bounces that would not trigger
for real bounces, too. A lot of those virus bounces ARE real bounces that
you would like to receive if you had in fact sent the message in question
(rather than some virus pretending to be you).

MSB

-- 
I used to be sane, but I got better.

On 28 Jan 2004, Michael Nolan wrote:
> At 08:11 27/01/2004, Jem wrote:
> 
> >I found spamprobe wasn't learning that these things were spam (they only
> >contain an attachment so they're isn't much data to go on). I'm sharing my
> >procmail rules I've derived by extracting arbitrary IDs from the various
> >strains I've received:
> >
> >http://wpbl.pc9.org/procmailrc
> >
> >In my rules, upon detection of the worm the message is dropped to the WPBL
> >data collection tool (this records the abusive IP and reports it to our
> >central server -- ends up on wpbl.dnsbl.net.au). If you're not a WPBL 
> >member
> >you will probably want to replace the findip call with /dev/null
> >
> 
> Wouldn't it be better just to use an virus scanner?  I'm using ClamAV
> (http://www.clamav.net/) and it works very well.  It downloaded updated
> virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.
> 
> I have it running from a Procmail rule along side SpamProbe without any
> problems.  Best of all, of course, is that it's free!
> 
> Mike
> 
> 
> --
> Michael Nolan
> michael@...
> http://www.michaelnolan.co.uk/


What procmail rule are you using for this?

Anthony

-- 
ac@...    ||  http://www.acampbell.org.uk
using Linux GNU/Debian ||  for book reviews, electronic 
Windows-free zone      ||  books and skeptical articles

--On Tuesday, January 27, 2004 7:57 AM -0800 Tim Smith <tzs@...> wrote:
> On Tue, 27 Jan 2004, Jem wrote:
>> I found spamprobe wasn't learning that these things were spam (they only
>> contain an attachment so they're isn't much data to go on). I'm sharing
>> my
>
> Hmmm...maybe spamprobe should consider lines that match the RE
> "^[a-zA-Z0-9+/]+$" to be a token?

Lack of content makes these sorts of things very difficult to catch.  They 
come from lots of different machines (including co-workers, friends, etc) 
so headers don't help much either.

SP is a spam detector rather than a virus detector.  It ignores binary 
attachments so that it doesn't bloat its database with lots of tokens that 
have nothing to do with spam.

It might be possible to try to create a few tokens from each binary 
attachment.  For example using an MD5 of the attachments bytes.  Not sure 
how helpful that would be though.  This worm doesn't but many do paste in 
text from the user's own email.  Good words from those would tend to 
overwhelm the two or three tokens from the attachment.

I installed Jem's procmail rules and they blocked the 280+ worm-mails that 
I received.  I also use a procmail rule to stop messages with empty bodies. 
Bounces tend to be blocked for me now.

All the best,
++Brian

--On Wednesday, January 28, 2004 9:54 AM +0000 Michael Nolan 
<michael@...> wrote:
> Wouldn't it be better just to use an virus scanner?  I'm using ClamAV
> (http://www.clamav.net/) and it works very well.  It downloaded updated
> virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.
>
> I have it running from a Procmail rule along side SpamProbe without any
> problems.  Best of all, of course, is that it's free!

Excellent!  I went looking for something like this but didn't find that 
one.  I'll give it a try.

All the best,
++Brian

if you have the cpu to do it, it's a very good idea to be running a
virus scanner :) i use sophos and amavis running through courierfilter
(a feature not unlike milter in sendmail): viruses don't even get
accepted in the mail queue! take that, nasties!


On Wed, 2004-01-28 at 04:54, Michael Nolan wrote:
> At 08:11 27/01/2004, Jem wrote:
> 
> >I found spamprobe wasn't learning that these things were spam (they only
> >contain an attachment so they're isn't much data to go on). I'm sharing my
> >procmail rules I've derived by extracting arbitrary IDs from the various
> >strains I've received:
> >
> >http://wpbl.pc9.org/procmailrc
> >
> >In my rules, upon detection of the worm the message is dropped to the WPBL
> >data collection tool (this records the abusive IP and reports it to our
> >central server -- ends up on wpbl.dnsbl.net.au). If you're not a WPBL member
> >you will probably want to replace the findip call with /dev/null
> >
> 
> Wouldn't it be better just to use an virus scanner?  I'm using ClamAV 
> (http://www.clamav.net/) and it works very well.  It downloaded updated 
> virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.
> 
> I have it running from a Procmail rule along side SpamProbe without any 
> problems.  Best of all, of course, is that it's free!
> 
> Mike
>

On 01/28, Michael Nolan wrote:
> Wouldn't it be better just to use an virus scanner?  I'm using ClamAV 
> (http://www.clamav.net/) and it works very well.  It downloaded updated 

Just so other's don't have to put unnecessary thought into this (very
useful solution):

Install clamav (http://www.clamav.net/). Under Debian, this will do it:
echo deb http://clamav.catt.com/debian stable main >> /etc/apt/sources.list;apt-get update;apt-get install clamav

Then add this to ~/.procmailrc:

VIRUS=`/usr/bin/clamscan --mbox --disable-summary --stdout -`
:0 fw:
| /usr/bin/formail -I "X-Virus: $VIRUS"
:0:
* ^X-Virus: .* FOUND
IN.viruses


-- 
What are you afraid of ?
http://www.ChaosReigns.com/

On Wed, 28 Jan 2004 12:02:13 +0000
Anthony Campbell <ac@...> wrote:

> > I have it running from a Procmail rule along side SpamProbe without
> > any problems.  Best of all, of course, is that it's free!
> > 
> > Mike
> 
> What procmail rule are you using for this?
> 
> Anthony
> 

This is indeed a good question. I've been fighting clamav to have a
result usable in procmail. Finally, I wrapped it in the following
script. I'm new to clamav. So maybe there is a more handsome solution.

-------------------------------------------
#!/bin/sh
#this script is called mailscan in the procmail rules
DIAG=`clamscan --mbox --stdout --disable-summary -i - `
if test "$?" == "0"
then
        echo "Passed"
else 
        echo $DIAG | sed -e 's/[^:]*: //'
fi
-------------------------------------------
Procmail uses mailscan as follows :
-------------------------------------------
:0 
AVOUT=| mailscan
:0 wf
| formail -I "X-ClamAV: $AVOUT"
:0 
* ^X-ClamAV:.*FOUND
Spam/
-------------------------------------------
In sane mails I got a nice header 
X-ClamAV: Passed
in viruses I got : 
X-ClamAV: WORM blabla FOUND

-- 
Cedric

"[Of course] I'm French! Why do think I have this outrageous 
accent, you silly king-a?!"  Monty Python and the Holy Grail

At 12:02 28/01/2004, Anthony Campbell wrote:

>On 28 Jan 2004, Michael Nolan wrote:
> > At 08:11 27/01/2004, Jem wrote:
> >
> > >I found spamprobe wasn't learning that these things were spam (they only
> > >contain an attachment so they're isn't much data to go on). I'm sharing my
> > >procmail rules I've derived by extracting arbitrary IDs from the various
> > >strains I've received:
> > >
> > >http://wpbl.pc9.org/procmailrc
> > >
> > >In my rules, upon detection of the worm the message is dropped to the WPBL
> > >data collection tool (this records the abusive IP and reports it to our
> > >central server -- ends up on wpbl.dnsbl.net.au). If you're not a WPBL
> > >member
> > >you will probably want to replace the findip call with /dev/null
> > >
> >
> > Wouldn't it be better just to use an virus scanner?  I'm using ClamAV
> > (http://www.clamav.net/) and it works very well.  It downloaded updated
> > virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.
> >
> > I have it running from a Procmail rule along side SpamProbe without any
> > problems.  Best of all, of course, is that it's free!
> >
> > Mike
> >
>
>
>What procmail rule are you using for this?
>
>Anthony

I've already sent this to Anthony, but just realised that he sent it to the 
list too, so here's my answer for everyone:

I'm using trashscan which I think is provided in the contrib directory for 
clamav.  The rules are:

:0
* multipart
* !^X-Virus-Scan:
| /usr/local/bin/trashscan

:0:
* ^X-Virus-Scan: Suspicious
virus


I follow this by my SpamProbe rules.

I'll add the following comments too.  This is probably the easiest way to 
get ClamAV up and running, but if you're using it with more than a couple 
of users, it would be better to have it plug into sendmail directly.

I have it check for updates every hour and it got signatures for MyDoom at 
around 10pm GMT on Monday night; several hours before the commercial AV 
companies.  I use AVG on my Windows machine which is great, but it's nice 
to have no viruses around when I login through webmail or under Linux.  I 
also get this feeling that it's risking it being too late to be stopping 
viruses after they're downloaded to the client and it's much better to get 
the server-side.

Hope this helps some people.

Mike

-- 
Michael Nolan
michael@...
http://www.michaelnolan.co.uk/

--On Wednesday, January 28, 2004 12:02 PM +0000 Anthony Campbell 
<ac@...> wrote:
> On 28 Jan 2004, Michael Nolan wrote:
>> Wouldn't it be better just to use an virus scanner?  I'm using ClamAV
>> (http://www.clamav.net/) and it works very well.  It downloaded updated
>> virus signatures to detect Mydoom at 22:00 GMT on 26th Jan.
>>
>> I have it running from a Procmail rule along side SpamProbe without any
>> problems.  Best of all, of course, is that it's free!
>
> What procmail rule are you using for this?

Here is what I used:

:0
VIRUS=| /usr/local/bin/clamscan --mbox --disable-summary --stdout -

:0 D
* $VIRUS ?? FOUND
{
  :0 wf
  | formail -I "X-ClamAV: $VIRUS"

  :0
  viral
}

I wanted to add a header with the virus description and save the original 
email.  If all you wanted to do was devnull viruses you could use something 
simpler like this:

:0
VIRUS=| /usr/local/bin/clamscan --mbox --disable-summary --stdout -

:0 D
* $VIRUS ?? FOUND
/dev/null


The assignment to a variable in that case is probably unnecessary.  I'm not 
a procmail guru so I'm not sure of the syntax to just run the command and 
test its output.

All the best,
++Brian

Is there something wrong with this procmail rule posted earlier today ?
I just put it in and so far it seems to be working ...


VIRUS=`clamscan --mbox --disable-summary --stdout -`
:0 fw
| formail -I "X-Virus: $VIRUS"

:0
* ^X-Virus: .* FOUND
Mail/Viruses



> On Wed, 28 Jan 2004 12:02:13 +0000
> Anthony Campbell <ac@...> wrote:
> 
> > > I have it running from a Procmail rule along side SpamProbe without
> > > any problems.  Best of all, of course, is that it's free!
> > > 
> > > Mike
> > 
> > What procmail rule are you using for this?
> > 
> > Anthony
> > 
> 
> This is indeed a good question. I've been fighting clamav to have a
> result usable in procmail. Finally, I wrapped it in the following
> script. I'm new to clamav. So maybe there is a more handsome solution.
> 
> -------------------------------------------
> #!/bin/sh
> #this script is called mailscan in the procmail rules
> DIAG=`clamscan --mbox --stdout --disable-summary -i - `
> if test "$?" == "0"
> then
>         echo "Passed"
> else 
>         echo $DIAG | sed -e 's/[^:]*: //'
> fi
> -------------------------------------------
> Procmail uses mailscan as follows :
> -------------------------------------------
> :0 
> AVOUT=| mailscan
> :0 wf
> | formail -I "X-ClamAV: $AVOUT"
> :0 
> * ^X-ClamAV:.*FOUND
> Spam/
> -------------------------------------------
> In sane mails I got a nice header 
> X-ClamAV: Passed
> in viruses I got : 
> X-ClamAV: WORM blabla FOUND
> 
> -- 
> Cedric
> 
> "[Of course] I'm French! Why do think I have this outrageous 
> accent, you silly king-a?!"  Monty Python and the Holy Grail
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Spamprobe-users mailing list
> Spamprobe-users@...
> https://lists.sourceforge.net/lists/listinfo/spamprobe-users
>

> > Wouldn't it be better just to use an virus scanner?
> 
> I agree. I see no way to Bayes filter false bounces that would not
> trigger for real bounces, too. A lot of those virus bounces ARE real
> bounces that you would like to receive if you had in fact sent the
> message in question (rather than some virus pretending to be you).

I don't have the resources to run a virus scanner. To date I have been using 
my own creation (renattach, GNU GPL) that catches all windows worms except 
for those within ZIP files. I intend to add that vital feature for the next 
version (searching for dangerous files inside ZIP).

Yes, the best solution is to use a real virus scanner. Detecting a virus/worm 
is definitely outside the scope of spamprobe!

Is there something wrong with this procmail rule posted earlier today ?
I just put it in and so far it seems to be working ...


VIRUS=`clamscan --mbox --disable-summary --stdout -`
:0 fw
| formail -I "X-Virus: $VIRUS"

:0
* ^X-Virus: .* FOUND
Mail/Viruses



> On Wed, 28 Jan 2004 12:02:13 +0000
> Anthony Campbell <ac@...> wrote:
> 
> > > I have it running from a Procmail rule along side SpamProbe without
> > > any problems.  Best of all, of course, is that it's free!
> > > 
> > > Mike
> > 
> > What procmail rule are you using for this?
> > 
> > Anthony
> > 
> 
> This is indeed a good question. I've been fighting clamav to have a
> result usable in procmail. Finally, I wrapped it in the following
> script. I'm new to clamav. So maybe there is a more handsome solution.
> 
> -------------------------------------------
> #!/bin/sh
> #this script is called mailscan in the procmail rules
> DIAG=`clamscan --mbox --stdout --disable-summary -i - `
> if test "$?" == "0"
> then
>         echo "Passed"
> else 
>         echo $DIAG | sed -e 's/[^:]*: //'
> fi
> -------------------------------------------
> Procmail uses mailscan as follows :
> -------------------------------------------
> :0 
> AVOUT=| mailscan
> :0 wf
> | formail -I "X-ClamAV: $AVOUT"
> :0 
> * ^X-ClamAV:.*FOUND
> Spam/
> -------------------------------------------
> In sane mails I got a nice header 
> X-ClamAV: Passed
> in viruses I got : 
> X-ClamAV: WORM blabla FOUND
> 
> -- 
> Cedric
> 
> "[Of course] I'm French! Why do think I have this outrageous 
> accent, you silly king-a?!"  Monty Python and the Holy Grail
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Spamprobe-users mailing list
> Spamprobe-users@...
> https://lists.sourceforge.net/lists/listinfo/spamprobe-users
>

I tried this solution. Unfortunately, with multipart message, clamav
will output one line for each part. So, if the virus is not the first
one, the rule ^X-Virus:.*FOUND may not work. You have to add -i flag,
which asks clamav to output only infected partl, but in this case,
X-Virus will be empty for non-infected file. Not very important...

I have a more detailed solution blocked in sourceforge smtp queue. It is
comming by foot, so please be patient :o)

On Wed, 28 Jan 2004 11:25:52 -0500
Darxus@... wrote:

> On 01/28, Michael Nolan wrote:
> > Wouldn't it be better just to use an virus scanner?  I'm using
> > ClamAV (http://www.clamav.net/) and it works very well.  It
> > downloaded updated 
> 
> Just so other's don't have to put unnecessary thought into this (very
> useful solution):
> 
> Install clamav (http://www.clamav.net/). Under Debian, this will do
> it: echo deb http://clamav.catt.com/debian stable main >>
> /etc/apt/sources.list;apt-get update;apt-get install clamav
> 
> Then add this to ~/.procmailrc:
> 
> VIRUS=`/usr/bin/clamscan --mbox --disable-summary --stdout -`
> :0 fw:
> | /usr/bin/formail -I "X-Virus: $VIRUS"
> :0:
> * ^X-Virus: .* FOUND
> IN.viruses
> 
> 
> -- 
> What are you afraid of ?
> http://www.ChaosReigns.com/
> 
> 
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> Spamprobe-users mailing list
> Spamprobe-users@...
> https://lists.sourceforge.net/lists/listinfo/spamprobe-users
> 


-- 
Cedric

"[Of course] I'm French! Why do think I have this outrageous 
accent, you silly king-a?!"  Monty Python and the Holy Grail

On Wed, Jan 28, 2004 at 08:39:35AM -0500, Brian Burton wrote:

> Lack of content makes these sorts of things very difficult to catch.  The=
y=20
> come from lots of different machines (including co-workers, friends, etc)=
=20
> so headers don't help much either.

Um. Heh. I think, for what it's worth, that SpamProbe has gotten each and
every one of these things on my system. Just a data point...

--=20
Mason Loring Bliss   mason@...   Oderint dum metuant!
https://bliss.dyndns.org/    awake ? sleep : random() & 2 ? dream : sleep;

> I tried this solution. Unfortunately, with multipart message, clamav
> will output one line for each part. So, if the virus is not the first
> one, the rule ^X-Virus:.*FOUND may not work. You have to add -i flag,
> which asks clamav to output only infected partl, but in this case,
> X-Virus will be empty for non-infected file. Not very important...

(a) The assignment should glob all the lines together in one big line.  

(b) Even if formail actually inserts continued lines in the message
    itself, procmail is smart enough to understand continued lines when
    you ask it to search for something in a header field.  (E.g., if
    there are a ton of people on the "To" line of a message and you are
    searching for someone who appears on the second or third line a
    pattern like "* ^To.*mallman" will still work just fine.

allman


--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

--On Thursday, January 29, 2004 11:13 AM +0100 Cedric Pradalier 
<cedric.pradalier@...> wrote:
> I tried this solution. Unfortunately, with multipart message, clamav
> will output one line for each part. So, if the virus is not the first
> one, the rule ^X-Virus:.*FOUND may not work. You have to add -i flag,
> which asks clamav to output only infected partl, but in this case,
> X-Virus will be empty for non-infected file. Not very important...

This isn't true if you assign the output of clamscan to a variable and then 
test that (as my procmail rule does).  I have received an email like this 
(first two parts OK, last two parts viral) and the procmail rule caught it 
and filed it to my viral folder.

All the best,
++Brian

On 02 Feb 2004, Brian Burton wrote:
> --On Thursday, January 29, 2004 11:13 AM +0100 Cedric Pradalier
> <cedric.pradalier@...> wrote:
> >I tried this solution. Unfortunately, with multipart message, clamav
> >will output one line for each part. So, if the virus is not the first
> >one, the rule ^X-Virus:.*FOUND may not work. You have to add -i flag,
> >which asks clamav to output only infected partl, but in this case,
> >X-Virus will be empty for non-infected file. Not very important...
> 
> This isn't true if you assign the output of clamscan to a variable and then
> test that (as my procmail rule does).  I have received an email like this
> (first two parts OK, last two parts viral) and the procmail rule caught it
> and filed it to my viral folder.
> 
> All the best,
> ++Brian

I think I must be missing something quite basic about this worm stuff.
I've installed clamav and various procmail rules that people have
suggested but nothing ever appears as having been filtered. Possibly
this is because I never receive any (?), since the mail server I use
claims to scan all the mail from viruses thus:
> ________________________________________________________________________
> This email has been scanned using the CleanPort MEF antivirus system.
> Funded for members by the Doctors.net.uk Bulletin service How does
> this protect me? http://www.Doctors.net.uk/qualityemail
> ________________________________________________________________________
> 

Nevertheless I still get occasional messages from hosts I don't know
telling me they have detected viruses in emails I haven't sent.
Presumably these are due to spammers who spoof my address. It's not a
great problem as far as I'm concerned but I'd prefer not to be sending
out these bogus emails if possible. 

Can anyone illuminate what is happening here? 

Anthony

-- 
ac@...    ||  http://www.acampbell.org.uk
using Linux GNU/Debian ||  for book reviews, electronic 
Windows-free zone      ||  books and skeptical articles

> Nevertheless I still get occasional messages from hosts I don't know
> telling me they have detected viruses in emails I haven't sent.
> Presumably these are due to spammers who spoof my address.

Yep.  That is exactly what you're seeing.  At times I have found this
backscatter to be worse than the virus.  The following procmail rule
came across nanog yesterday (from Ramdy Bush).  It looks reasonable to
me and I have installed it in an attempt to counter the backscatter.  I
briefly review the folder these things are dropped into daily for false
positives.  But, since I just added this rule yesterday I don't have
much operational experience to share.  So, YMMV.

## try to catch virus backscatter (from Randy Bush)

:0
* ^Subject:[ ](\{Spam\?\} Warning: E-mail viruses detected|Anti-Virus Notification|BANNED FILENAME|Disallowed attachment type found in sent message|File blocked - ScanMail for Lotus|InterScan NT Alert|Message delete|NAV detected a virus|Norton AntiVirus detected|RAV AntiVirus scan|Symantec AntiVirus|Undeliverable: test|VIRUS \(.*\) IN MAIL (FROM|TO) YOU|VIRUS IN YOUR MAIL|Virus Detected by Network Assoc|Virus Notification|Virus found in a message you sent|Virus found in sent message)
ALTERNATE/Junk/virii-junk


allman


--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

On 03 Feb 2004, Mark Allman wrote:
> 
> > Nevertheless I still get occasional messages from hosts I don't know
> > telling me they have detected viruses in emails I haven't sent.
> > Presumably these are due to spammers who spoof my address.
> 
> Yep.  That is exactly what you're seeing.  At times I have found this
> backscatter to be worse than the virus.  The following procmail rule
> came across nanog yesterday (from Ramdy Bush).  It looks reasonable to
> me and I have installed it in an attempt to counter the backscatter.  I
> briefly review the folder these things are dropped into daily for false
> positives.  But, since I just added this rule yesterday I don't have
> much operational experience to share.  So, YMMV.
> 
> ## try to catch virus backscatter (from Randy Bush)
> 
> :0
> * ^Subject:[ ](\{Spam\?\} Warning: E-mail viruses detected|Anti-Virus Notification|BANNED FILENAME|Disallowed attachment type found in sent message|File blocked - ScanMail for Lotus|InterScan NT Alert|Message delete|NAV detected a virus|Norton AntiVirus detected|RAV AntiVirus scan|Symantec AntiVirus|Undeliverable: test|VIRUS \(.*\) IN MAIL (FROM|TO) YOU|VIRUS IN YOUR MAIL|Virus Detected by Network Assoc|Virus Notification|Virus found in a message you sent|Virus found in sent message)
> ALTERNATE/Junk/virii-junk
> 
> 
> allman
> 
> 
> --
> Mark Allman -- ICIR -- http://www.icir.org/mallman/
> 

I'll give this a try.

Anthony

-- 
ac@...    ||  http://www.acampbell.org.uk
using Linux GNU/Debian ||  for book reviews, electronic 
Windows-free zone      ||  books and skeptical articles

On 02/03/2004 02:43 PM, Anthony Campbell wrote:

>Nevertheless I still get occasional messages from hosts I don't know
>telling me they have detected viruses in emails I haven't sent.
>Presumably these are due to spammers who spoof my address. It's not a
>great problem as far as I'm concerned but I'd prefer not to be sending
>out these bogus emails if possible. 
>
It's not spammers that use your e-mail address. It's thanks to the 
inventions of the clever virus makers that this happens. Not only do 
virusses spread to everyone in the average Outlook address-book. They 
also scan the e-mails and hunt for e-mail addresses there. And often, 
one is picked which then is used as the sender address. So if someone 
running windows and outlook gets a virus and (s)he has your e-mail 
address somewhere, you have the chance that your address is used as the 
spreader of the virus.

Been there, believe me. Apparently, on my Mandrake machine, I am running 
The Bat! and I don't know how many versions of Outlook etc, if I can 
believe the returned mails...

Paul

-- 
Swords and guns have no eyes.
-Chinese proverb

http://www.nlpagan.net - Linux Mandrake - Thunderbird 0.5a/20031217

Kurt has been trying my smtp filter and I've made a lot of progress with
his feedback.  If anyone else is in a position to *alpha* test this
code, I would be greatful.

The latest release now does the following:

1) uses spamassassin to auto-train spamprobe
2) uses clamav to catch viruses
 [both can be removed by commenting out a #define ]

It has one bug which I've been too busy to fix - a broken smtp mailer
which sends only a \r in certain contexts causes a problem; I've
bodged it with an alarm() call as a belt-and-suspenders mechanism to
protect against program failures.  For the moment I am not fixing the
bug and leaving that in, to test the robustness of the program in the
face of unexpected behaviour such as you might get from broken MTAs.

Despite the warning above, it does work and for me it is working very
nicely in two production environments.

Don't test this on a live system though.  If you have a spare machine
which you can redirect a copy of mail to from some other system, that
would be the safest way to test it.

[Do we have any unix system gurus here?  I'm looking for someone who
can help me 'productize' this (apologies for using that word) by building
it into a transparent bridge.]

thanks

Graham
PS, Brian: I have seen the light :-) ... I realised that the tokenizing stuff
I was suggesting is ideal *for header tests*, but that it may still be
better to leave the body to traditional tokens.  I already wrote code
for filtering based on static strings in the headers.  All I need to
do is tweak that a little to make them dynamic.

Folks, I have the worlds weirdest problem.  Can anyone see anything
obviously wrong with this?  I have started getting the same mail
delivered multiple times.  There's two sets of logs below - the first
was mail to me, and it appears to have been sent and received OK, but
it keeps coming.  The second was a mail to one of my users, and for 
some reason the sender sent a RSET after it was delivered, and he too
kept on reconnecting.  What's more the last one retried  a *lot* ...
I've trimmed the log down as there are *hundreds* more entries just like
that one...  and those are only two of several emails showing this problem.

I've stared at this until I'm word-blind.  Is there something obvious
here I'm missing???  The log itself is fairly trustworthy - I'm sure that
if it is in the log, the client saw it as well.

This is not happening on every mail, just a few specific ones. And it
happens to both email and spam.

At the end is an almost-complete session of each (only a little
body content elided for privacy reasons).

If it weren't happening on so many mails I'd just think it was a network
error, but it must be something in my software.  I just can't see it.

It *looks* like my code is obeying the RFC and it is the sender that is
misbehaving, but that would *not* happen from so many sites at once.  It
just isn't plausible.

Any ideas?

thanks

G

(just the last few lines of each of the mails...)

==> 15667.log <==
[t=2]WOMC Support Department \r\n
[t=2].\r\n
[t=3]250 2.0.0 i13MMZdQ015668 Message accepted for delivery
[t=3]QUIT\r\n
[t=3]221 2.0.0 gtoal.com closing connection

==> 15812.log <==
[t=2]WOMC Support Department \r\n
[t=2].\r\n
[t=2]250 2.0.0 i13MNcdQ015813 Message accepted for delivery
[t=2]QUIT\r\n
[t=2]221 2.0.0 gtoal.com closing connection

==> 15927.log <==
[t=2]WOMC Support Department \r\n
[t=2].\r\n
[t=2]250 2.0.0 i13MOedQ015928 Message accepted for delivery
[t=3]QUIT\r\n
[t=3]221 2.0.0 gtoal.com closing connection

==> 15968.log <==
[t=2]WOMC Support Department \r\n
[t=2].\r\n
[t=3]250 2.0.0 i13MPjdQ015969 Message accepted for delivery
[t=3]QUIT\r\n
[t=3]221 2.0.0 gtoal.com closing connection

--------------------------------------

==> 10106.log <==
[t=2].\r\n
[t=3]250 2.0.0 i141fmdQ010107 Message accepted for delivery
[t=3]RSET\r\n
[t=3]250 2.0.0 Reset state
[t=3][eof t=3]
==> 11466.log <==
[t=2].\r\n
[t=2]250 2.0.0 i13LljdQ011467 Message accepted for delivery
[t=2]RSET\r\n
[t=2]250 2.0.0 Reset state
[t=2][eof t=2]
==> 11859.log <==
[t=1].\r\n
[t=2]250 2.0.0 i141nmdQ011860 Message accepted for delivery
[t=2]RSET\r\n
[t=2]250 2.0.0 Reset state
[t=2][eof t=2]
==> 12659.log <==
[t=2].\r\n
[t=2]250 2.0.0 i13M0mdQ012660 Message accepted for delivery
[t=2]RSET\r\n
[t=2]250 2.0.0 Reset state
[t=3][eof t=3]
==> 12824.log <==
[t=1].\r\n
[t=2]250 2.0.0 i13M1kdQ012825 Message accepted for delivery
[t=2]RSET\r\n
[t=2]250 2.0.0 Reset state
[t=2][eof t=2]

-------------------------------
[t=0]220 gtoal.com ESMTP Sendmail 8.12.10/8.12.10/SuSE Linux 0.7; Tue, 3 Feb 2004 16:22:35 -0600 +SPAMFILTER 0.5.4
[t=0]EHLO knud\r\n
[t=0]250-gtoal.com Hello localhost [127.0.0.1], pleased to meet you
[t=0]250-ENHANCEDSTATUSCODES
[t=0]250-PIPELINING
[t=0]250-8BITMIME
[t=0]250-SIZE
[t=0]250-DSN
[t=0]250-ETRN
[t=0]250-DELIVERBY
[t=0]250 HELP
[t=1]MAIL FROM:<notification.system@...
[t=1]250 2.1.0 <notification.system@...>... Sender ok
[t=2]RCPT TO:<gtoal@...
[t=2]250 2.1.5 <gtoal@...>... Recipient ok
[t=2]DATA\r\n
[t=2]354 Enter mail, end with "." on a line by itself
[t=2]Received: from henning ([127.0.0.1]) by knud with Microsoft SMTPSVC(5.0.2195.6713);\r\n
[t=2]	 Tue, 3 Feb 2004 14:15:28 -0800\r\n
[t=2]Date: Tue, 03 Feb 2004 14:15:28 -0800\r\n
[t=2]From: notification.system@...
[t=2]Subject: A user is researching your background\r\n
[t=2]To: gtoal@...
[t=2]Return-Path: notification.system@...
[t=2]Message-ID: <HENNINGQPqgy9gSXUPZ000095e0@...
[t=2]X-OriginalArrivalTime: 03 Feb 2004 22:15:28.0581 (UTC) FILETIME=[3AEEE750:01C3EAA3]\r\n
[t=2]\r\n
[t=2]One of our users just began researching your background at our website\r\n
[t=2]\r\n
[t=2]As a result, our website has automatically sent you this email to make you aware of this.\r\n
[t=2]\r\n
[t=2]If another person begins to research you in the future our website will again inform you via email.\r\n
[t=2]\r\n
[t=2]You may view the research being done via:\r\n
[t=2]http://s.wominfo.info/pass.php?a=search&b=5&c=gtoal@...>
[t=2]\r\n
[t=2]You can easily avoid receiving further email from our website by adding your email address to our Do Not Email List via:\r\n
[t=2]http://s.wominfo.info/pass.php?a=donotemail&b=gtoal@...>
[t=2]\r\n
[t=2]Regards,\r\n
[t=2]\r\n
[t=2]WOMC Support Department \r\n
[t=2].\r\n
[t=3]250 2.0.0 i13MMZdQ015668 Message accepted for delivery
[t=3]QUIT\r\n
[t=3]221 2.0.0 gtoal.com closing connection

-------------------------------------------------------------

[t=0]220 gtoal.com ESMTP Sendmail 8.12.10/8.12.10/SuSE Linux 0.7; Tue, 3 Feb 2004 19:49:48 -0600 +SPAMFILTER 0.6.1
[t=0]EHLO private.org\r\n
[t=0]250-gtoal.com Hello localhost [127.0.0.1], pleased to meet you
[t=0]250-ENHANCEDSTATUSCODES
[t=0]250-PIPELINING
[t=0]250-8BITMIME
[t=0]250-SIZE
[t=0]250-DSN
[t=0]250-ETRN
[t=0]250-DELIVERBY
[t=0]250 HELP
[t=0]MAIL From:<PRIVATE@...
[t=0]250 2.1.0 <PRIVATE@...>... Sender ok
[t=0]RCPT To:<private@...
[t=0]250 2.1.5 <private@...>... Recipient ok
[t=1]DATA\r\n
[t=1]354 Enter mail, end with "." on a line by itself
[t=1]Received: by private.org id <119111>; Tue, 3 Feb 2004 13:40:32 -0600\r\n
[t=1]Message-Id: <04Feb3.134032cst.119111@...
[t=1]X-Mailer: Novell GroupWise Internet Agent 6.5.1 \r\n
[t=1]Date:   Tue, 03 Feb 2004 13:40:04 -0600\r\n
[t=1]From: "Private Private" <PRIVATE@...
[t=1]To: <PRIVATE@...>, <PRIVATE@...
[t=1]Subject: Re: FW: Private Private\r\n
[t=1]Mime-Version: 1.0\r\n
[t=1]Content-Type: text/plain; charset=US-ASCII\r\n
[t=1]Content-Transfer-Encoding: 7bit\r\n
[t=1]Content-Disposition: inline\r\n
[t=1]\r\n

... normal looking content deleted for privacy ...

[t=1]        Last line of message.\r\n
[t=1].\r\n
[t=2]250 2.0.0 i141nmdQ011860 Message accepted for delivery
[t=2]RSET\r\n
[t=2]250 2.0.0 Reset state
[t=2][eof t=2]

On 03 Feb 2004, Paul wrote:
> On 02/03/2004 02:43 PM, Anthony Campbell wrote:
> 
> >Nevertheless I still get occasional messages from hosts I don't know
> >telling me they have detected viruses in emails I haven't sent.
> >Presumably these are due to spammers who spoof my address. It's not a
> >great problem as far as I'm concerned but I'd prefer not to be sending
> >out these bogus emails if possible.
> >
> It's not spammers that use your e-mail address. It's thanks to the
> inventions of the clever virus makers that this happens. Not only do
> virusses spread to everyone in the average Outlook address-book. They
> also scan the e-mails and hunt for e-mail addresses there. And often,
> one is picked which then is used as the sender address. So if someone
> running windows and outlook gets a virus and (s)he has your e-mail
> address somewhere, you have the chance that your address is used as the
> spreader of the virus.
> 
> Been there, believe me. Apparently, on my Mandrake machine, I am running
> The Bat! and I don't know how many versions of Outlook etc, if I can
> believe the returned mails...
> 
> Paul
> 

I inserted the procmail rule which Mark posted here recently. It seems
to work, in the sense that I got the following in my virus folder:

================================================================
Date: 4 Feb 2004 05:47:50 -0000
>From postmaster@...  Wed Feb  4 05:49:16 2004
From: System Anti-Virus Administrator <postmaster@...>
Subject: virus found in sent message ""

[rest of heading snipped]

Attention: tom@...

A virus was found in an Email message you sent. 
This Email scanner intercepted it and stopped the entire message
reaching its destination. 

The virus was reported to be: 

 W32/Mydoom.A@...

Please update your virus scanner or contact your IT support 
personnel as soon as possible as you have a virus on your system.
========================================================

Needless to say, there is no Tom at this address. It appears that the
procmail rule catches the return message (which I suppose spamprobe
could also do, with training) but this is bolting the computer door
after the virus has bolted. In other words I'm still sending out the
offending emails and helping to clog up the Internet but am just filing
the return messages in an appropriate folder. Not a great deal of help
really.

Anthony


-- 
ac@...    ||  http://www.acampbell.org.uk
using Linux GNU/Debian ||  for book reviews, electronic 
Windows-free zone      ||  books and skeptical articles

On 02/04/2004 11:21 AM, Anthony Campbell wrote:

>after the virus has bolted. In other words I'm still sending out the
>offending emails and helping to clog up the Internet
>  
>
It may look like you are doing it. But you know (I assume) that you are 
not doing it. No need to hit yourself over the head over something that 
a virus in an unknown computer does while using (part of) your name.

Windoze virae do not perform well in a Linux environment...

Paul

-- 
I believe in God,
Only I spell it Nature.
-Frank Lloyd Wright

http://www.nlpagan.net - Linux Mandrake - Thunderbird 0.5a/20031217

> Windoze virae do not perform well in a Linux environment...

Except for wasting our bandwidth... they're still pretty darn good at doing 
that. Maybe it would have been a good thing if the MS/Outlook/Exchange world 
kept separate from the open standard SMTP world. Then spammers and worms 
would predominantly be a Microsoft problem, and we simply wouldn't accept 
mail transactions on the ms-exchange TCP port :)

Graham,

Sending a RSET after a mail has been sent successfully is not unusual.
The connecting MTA probably wants to send another mail on the same
connection (pipelining). The RSET is used to make sure the state is
reset before beginning the transaction for the next message.

What would be of major interest is which end is actively closing the TCP
socket (or does it stay open?). I did a quick test on your host and I
was not disconnected after sending a mini-message and then doing a RSET.
Maybe you can shed some light onto it if you check what MTA is being
used on the other end; might be a problem there.

Why the same message is sent over and over again is unclear to me,
though. After receiving the 250 Messge Accepted, it should be purged
from the spool.

Kurt


On Tue, Feb 03, 2004 at 11:19:12PM -0600, Graham Toal wrote:
> ...
> it keeps coming.  The second was a mail to one of my users, and for 
> some reason the sender sent a RSET after it was delivered, and he too
> kept on reconnecting.  What's more the last one retried  a *lot* ...
> ...

> Folks, I have the worlds weirdest problem.  Can anyone see anything
> obviously wrong with this?  I have started getting the same mail
> delivered multiple times.  There's two sets of logs below ...

And the moral of this story is, look at it for yourself and don't
trust the logs :-)

There was nothing wrong with the code except carelessness on my part -
I built and tested it on one system, then moved it to another 'in
production'...  unfortunately I had forgotten to install clamav on the
second system, and when it was invoked from popen("clamscan", ...)
the shell gave an error to stderr.  Which the client saw, while
expecting to see a 2XX return code!  And of course I was only logging
what *I* wrote to the client so the error never made it into my logs.

Blech, three days wasted.  Time now for some ultra-defensive programming!

Kurt, I was saying to hold off until I'd found the problem: you can
use it now.  Just make sure spamassassin and clamav are installed, or
comment out the #define while enables them.

()()
  b=======

(that's a rather poor ascii version of me kicking myself in the butt...)

By the way, it was interesting to see how few mailers actually
cared that the return code was broken.  Of the ones who did, several
of them were spams - which probably means they are open relays
because I doubt very much that spammer tools would retry over a
period of days! ...  that gives me an idea of a new way to detect
open relays with a honeypot... hmmm...

G