dspam-devel — Developer, new features and building mailinglist

[Dspam-devel] Odd shared library-global variable problem on Mac OS X?

[Jesse P. <je...@ma...>]

Hello,

After having generally good success with preliminary testing with dspam I've run into an issue that after a few hours of delving deeper on appears to either be a bug or a platform support issue.

I'm compiling on Mac OS X 10.6.8 (old, I know) with the x86_64 architecture and am having a weird issue with the preferences C module (I think). Version 3.10.2. Here's what I'm seeing:

When executing dspam from my MTA (postfix) using the "LDA proxy" method dspam won't accept the user's preferences because it complains they're "disallowed". Like so:


dspam[35366]: Ignoring disallowed preference 'trainingMode'
dspam[35366]: Ignoring disallowed preference 'spamAction'
dspam[35366]: Ignoring disallowed preference 'spamSubject'
dspam[35366]: Ignoring disallowed preference 'statisticalSedation'
dspam[35366]: Ignoring disallowed preference 'enableBNR'
dspam[35366]: Ignoring disallowed preference 'optIn'
dspam[35366]: Ignoring disallowed preference 'optOut'
dspam[35366]: Ignoring disallowed preference 'showFactors'
dspam[35366]: Ignoring disallowed preference 'enableWhitelist'
dspam[35366]: Ignoring disallowed preference 'signatureLocation'
dspam[35366]: Ignoring disallowed preference 'dailyQuarantineSummary'


Now one's first reaction to this would be: "well, of course, you don't have any AllowOverrides in your dspam.conf" to which I'd say, no, actually I do. In fact this is the default (or at least very little modified) dspam.conf as installed by the Makefiles (which include the AllowOverrides). Yes, I've checked many times: they are there, I assure you.

So after checking that about a million times (since that seemed to be the issue) I started looking further. I eventually compiled with verbose logging, turned on logging, and reviewed the logs. This surprised me:


32189: [03/14/2013 20:00:39] Loading preferences from dspam.conf
32189: [03/14/2013 20:00:39] match attribute 'AllowOverride' with value 'trainingMode'
32189: [03/14/2013 20:00:39] find attribute 'AllowOverride'
32189: [03/14/2013 20:00:39] _ds_find_attribute(): NULL config
32189: [03/14/2013 20:00:39]  -> match: not found attribute 'AllowOverride'
32189: [03/14/2013 20:00:39] match attribute 'AllowOverride' with value 'spamAction'
32189: [03/14/2013 20:00:39] find attribute 'AllowOverride'
32189: [03/14/2013 20:00:39] _ds_find_attribute(): NULL config
32189: [03/14/2013 20:00:39]  -> match: not found attribute 'AllowOverride'


This was quite odd. Then looking at the code, specifically src/pref.c's _ds_pref_aggregate() (around line 55) which leads to src/config_shared.c _ds_find_attribute() (around line 47) where these errors emitted.

So it appears as though the global variable agent_config is NULL when pref.c does the preference "aggregation" runs. THe code's logic essentially pretends that that means there is no AllowOverrides and thus preferences aren't used.

So, not quite sure what to do with this on a hunch I compiled using static libraries (configured with --disabled-shared) and viola - dspam respected my user preferences. Presumably it was able to see the agent_config variable. I haven't verified but I think dspam (where agent_config is set) sets agent_config loads libdspam which has _ds_pref_aggregate().

I think, but am not sure, that's what this thread is about, too, as it is also on Mac OS X (macports):

	http://sourceforge.net/mailarchive/message.php?msg_id=29506400

I'm willing to help track this down further if folks could point me in the right direction. Happy to file a bug if wanted, too.

Thoughts?


Thanks,
- Jesse

[Dspam-devel] [ dspam-Bug Tracker-3607521 ] Active loop in daemon could be avoided

[SourceForge.net <no...@so...>]

Bug Tracker item #3607521, was opened at 2013-03-10 05:51
Message generated for change (Tracker Item Submitted) made by glandos
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3607521&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Glandos (glandos)
Assigned to: Nobody/Anonymous (nobody)
Summary: Active loop in daemon could be avoided

Initial Comment:
Currently, the daemon has an active loop. It calls select with a 2 seconds timeout, just to to be sure that no signal was raised.
This could be avoided. I choose the self-pipe trick, cause it's the solution that integrate the most with the current code.
Using pselect would have meant blocking signals just before the call, and would have needed a re-enabling in the process request. I didn't have the time to appreciate the impact of that.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3607521&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3536088 ] dot stuffing bug

[SourceForge.net <no...@so...>]

Bug Tracker item #3536088, was opened at 2012-06-18 11:27
Message generated for change (Comment added) made by mozillion
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3536088&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: None
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: polaris1 (polaris1)
Assigned to: Nobody/Anonymous (nobody)
Summary: dot stuffing bug

Initial Comment:
I tried this in both 3.9.0 and in 3.10.1 with similar results.

I run dspam in client/daemon mode ( daemon is 3.10.1, clients I tried both 3.9.0 and 3.10.1 )

DSPAM is invoked from C with these arguments: dspam --client --user myuser --deliver=innocent,spam --stdout while the message is being fed in through stdin.

I have a copy of the message and the relevant part is here:

--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
.

Hello World, how are you


quoted printable only allows 76 characters per line and as it happens, that line ends with a dot on the 77th character thus putting it on a new line ( what are the odds, right ? )

situation can be reproduced like this: cat messagefile | dspam --client --user myuser --deliver=innocent,spam --stdout

For that message, the output will stop at 'next week to you='.

If I modify the message to stuff the dot manually (..) it will output:


--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
..

Hello World, how are you


Obviously something is wrong. I see that dspam has dot stuffing mechanisms in client.c but it's obviously not working. Is it because of the quoted printable ?

----------------------------------------------------------------------

Comment By: Paul van Tilburg (mozillion)
Date: 2013-02-25 07:32

Message:
I wonder what --client really should have to do with dot stuffing?  I mean,
--client is just about how it is handle in the backend (either run a sole
instance or use the client/daemon model). 

In my case I use a pipe to forward the (not dot-stuffed message) in
combination with --client for scalability reasons, but then suddenly, dot
stuffing is not applied and the message gets truncated.  Without --client
it all works fine.  I find this weird and, I hope we can agree on this, at
least undocumented behaviour.

----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 17:18

Message:
When running with --stdout, it's obvious that there should not be any dot
stuffing happening, so maybe this behavior is ok after all.

The question is, why is DSPAM concerned about dots as a message end marker,
when the daemon replies.

In daemon.c we have:

    /* Send a terminating '.' if --stdout in 'dspam' mode */

    if (ATX->sockfd_output) {
      if (send_socket(TTX, ".")<=0)
        goto CLOSE;

and in client.c there is

while(line != NULL && strcmp(line, ".")) {

Why can't that code be removed entirely from daemon.c and the line in
client.c turned into:

while(line != NULL) {

It would take care of the problem it seems. What am I missing ?

----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 14:54

Message:
So yeah, I think we can safely call this a bug now.

1. dspam client properly does dot stuffing
2. dspam daemon properly does dot de-stuffing when analyzing the message
3. dspam daemon does not re-stuff the dot when sending the reply back to
the client

in client.c we have this line:
while(line != NULL && strcmp(line, ".")) {

which automatically cuts the message off at the first lone dot, unprocessed
by #3 above

----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 14:40

Message:
Yeah, it's definitely within the daemon code where there is no dot stuffing
code.

Using TCPDUMP here's the output of the outgoing packets towards the
daemon:


Dear Daniela,

According to Michela, I will send the loading schedule for next week to
you=
..

Hello World, how are you
.


and Incoming from Daemon to Client:


Dear Daniela,

According to Michela, I will send the loading schedule for next week to
you=
.

Hello World, how are you
.


It fails to stuff it properly on the way back...

----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 14:30

Message:
Furthermore, I added this code in client.c in order to make sure the dot
stuffing code is working:

r = send(TTX.sockfd, buf+t, buflen - t, 0);
      strncpy(sbuf, buf+t, r);
      sbuf[r]='\0';
      printf ("SENT: %s\n\n\n", sbuf);

And it prints properly the following:


SENT: "_005_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_";
        type="multipart/alternative"
MIME-Version: 1.0

--_005_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: multipart/alternative;
       
boundary="_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_"

--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to
you=
..

Hello World, how are you

So obviously the client function detects the lone dot, stuffs it and sends
it over properly. There has to be a bug in the daemon code ?


----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 12:49

Message:
There is no MTA involved here. dspam in --client mode, according to my
interpretation of client.c should perform dot stuffing.

If I read that incorrectly, can you tell me what all that code I pasted in
client.c is supposed to do ?

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2012-06-18 12:43

Message:
Yes. DSPAM binary is responsible doing the dot stuffing WHEN sending to a
delivery agent OR sending to the DSPAM daemon.

But in your case you feed the DSPAM binary with wrong data. So DSPAM stops
processing the message after you just send ONE dot.

It is the same as if you would open a connection to port 25 on your system
and post something like this:

< 220 mail.example.com ESMTP Postfix (2.9.3)\r
> EHLO mail.example.com\r
< 250-mail.example.com\r
< 250-PIPELINING\r
< 250-SIZE 52428800\r
< 250-ETRN\r
< 250-STARTTLS\r
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5\r
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5\r
< 250-ENHANCEDSTATUSCODES\r
< 250-8BITMIME\r
< 250 DSN\r
> MAIL FROM:<ro...@ex...>\r
< 250 2.1.0 Ok\r
> RCPT TO:<st...@ba...>\r
< 250 2.1.5 Ok\r
> DATA\r
< 354 End data with <CR><LF>.<CR><LF>\r
> Return-Path: <ro...@ex...>\r
> Delivered-To: st...@ba...\r
> To: <st...@ba...>\r
> From: <ro...@ex...>\r
> MIME-Version: 1.0\r
> Content-Type: multipart/related;\r
>        
boundary="_004_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_";\r
>         type="multipart/alternative"\r
> Subject: Testing\r
> \r
> --_004_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_\r
>        
boundary="_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_"\r
> \r
> --_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_\r
> Content-Type: text/plain; charset="us-ascii"\r
> Content-Transfer-Encoding: quoted-printable\r
> \r
> Dear Daniela,\r
> \r
> According to Michela, I will send the loading schedule for next week to
you=\r
> .\r
> \r
< 250 2.0.0 Ok: queued as 7FDC0A3C170\r
> Hello World, how are you\r
< 502 5.5.2 Error: command not recognized\r
> RSET\r
< 250 2.0.0 Ok\r
> QUIT\r
< 221 2.0.0 Bye\r


Assuming you run Postfix, would you say that this is a Postfix bug? I
personally would say that this is NOT a bug.


Now on the other hand if you have a MTA that is getting the above mail and
then sends that message to DSPAM over SMTP then the MTA is responsible to
do the dot stuffing, while DSPAM is responsible for doing the dot
destuffing. When the message is inside DSPAM (the client) and the client is
sending the data to the DSPAM daemon (because of the use of --client) then
the DSPAM client is doing dot stuffing while the DSPAM daemon is doing the
dot destuffing.




----------------------------------------------------------------------

Comment By: polaris1 (polaris1)
Date: 2012-06-18 12:06

Message:
Hmm, I saw this in client.c. I thought DSPAM client would take care of
that:

 * DESCRIPTION
 *   connect to a dspam daemon socket and attempt to process a message
 *   this function is called by the dspam agent when --client is specified


 /*
     * fill buf with partial msg, replacing \n with \r\n
     * and do dot stuffing, if needed.
     */


/* take care of dot stuffing \n */
          if (message->data[i + 1] && message->data[i + 1] == '.') {
            buf[buflen] = '\n';
            buflen++;
            buf[buflen] = '.';
            buflen++;
            buf[buflen] = '.';
            buflen++;
            i += 2;
            continue;
          }


I haven't really debugged the code, but it seems there is code there that
is supposed to do dot stuffing...

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2012-06-18 11:54

Message:
I am not really sure if this is a bug or not. Actually the application
sending data to DSPAM is responsible for doing the dot stuffing. In your
case you use cat to pipe the message to DSPAM. So 'cat' should do the dot
stuffing. Correct me if you see that differently.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3536088&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3605628 ] Unsafe use of strcpy in dspam.c

[SourceForge.net <no...@so...>]

Bug Tracker item #3605628, was opened at 2013-02-22 01:22
Message generated for change (Tracker Item Submitted) made by robotux
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605628&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: GIT
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Thomas Preud'homme (robotux)
Assigned to: Nobody/Anonymous (nobody)
Summary: Unsafe use of strcpy in dspam.c

Initial Comment:
Hi Stevan,

while trying to determine if commit cbed19764df8ecc1469f739ca33eb92aa3f67584 is overflow safe, I noticed an unsafe use of strcpy in dspam.c, (at line 1701 at commit f6c5008357248cf357b6666c7a6a4618fcf52d89).

If the domain name is longer than the extension, then strcpy is made on overlapping memory which is not guaranteed to work. AFAIK, it should fail with recent glibc since they take advantage of the fact areas should not overlap on strcpy to make more optimizations. Memmove should be used instead.

Best regards,

Thomas

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605628&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3605197 ] problem with email retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3605197, was opened at 2013-02-18 07:42
Message generated for change (Comment added) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605197&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Nobody/Anonymous (nobody)
Summary: problem with email retraining

Initial Comment:
Good morning,
I configured postfix to send missed spam mail to dspam through pipe with the following entry in the master.cf configuration file:

spam   unix    -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/local/bin/dspamc --user root --class=spam --source=error

It seems that the system works fine but in the maillog i have the following error:

Feb 18 16:06:54 dspam dspam[19530]: Signature retrieval for '122,512239e722133388817095' failed
Feb 18 16:06:54 dspam dspam[19530]: Unable to find a valid signature. Aborting.
Feb 18 16:06:54 dspam dspam[19530]: process_message returned error -5.  dropping message.

If I make an sql query on DB, i can find a record with signature = '122,512239e722133388817095'

what was wrong?

Thanks

----------------------------------------------------------------------

>Comment By: https://www.google.com/accounts ()
Date: 2013-02-19 10:55

Message:
I can see that the problem is that I used --user root calling dspamc. If i
use the correct username that "possesses" the signature all is fine. I have
resolved using a perl script, instead of the direct pipe to dspam, that
finds the username relative to the signature and then calls dspam with the
correct user. thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605197&group_id=250683

[Dspam-devel] [PATCH] Fix building modules

[Andreas S. <as...@cr...>]

Hey,

here is a patch to build modules correctly. rpmlint detected them:


[  137s] dspam-backend-hash.x86_64: E: devel-file-in-non-devel-package 
(Badness: 50) /usr/lib64/dspam/libhash_drv.so
[  137s] dspam-backend-pgsql.x86_64: E: devel-file-in-non-devel-package 
(Badness: 50) /usr/lib64/dspam/libpgsql_drv.so
[  137s] dspam-backend-sqlite3.x86_64: E: devel-file-in-non-devel-package 
(Badness: 50) /usr/lib64/dspam/libsqlite3_drv.so
[  137s] dspam-backend-mysql.x86_64: E: devel-file-in-non-devel-package 
(Badness: 50) /usr/lib64/dspam/libmysql_drv.so
[  137s] A development file (usually source code) is located in a non-devel 
package. If
[  137s] you want to include source code in your package, be sure to create a
[  137s] development package.


-- 
Andreas Schneider                   GPG-ID: F33E3FC6
www.cryptomilk.org                as...@cr...

[Dspam-devel] [ dspam-Bug Tracker-3605201 ] problem with email retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3605201, was opened at 2013-02-18 08:05
Message generated for change (Settings changed) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605201&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
>Status: Deleted
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Nobody/Anonymous (nobody)
Summary: problem with email retraining

Initial Comment:
Good morning,
I configured postfix to send missed spam mail to dspam through pipe with the following entry in the master.cf configuration file:

spam   unix    -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/local/bin/dspamc --user root --class=spam --source=error

It seems that the system works fine but in the maillog i have the following error:

Feb 18 16:06:54 dspam dspam[19530]: Signature retrieval for '122,512239e722133388817095' failed
Feb 18 16:06:54 dspam dspam[19530]: Unable to find a valid signature. Aborting.
Feb 18 16:06:54 dspam dspam[19530]: process_message returned error -5.  dropping message.

If I make an sql query on DB, i can find a record with signature = '122,512239e722133388817095'

what was wrong?

Thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605201&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3605201 ] problem with email retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3605201, was opened at 2013-02-18 08:05
Message generated for change (Tracker Item Submitted) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605201&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Nobody/Anonymous (nobody)
Summary: problem with email retraining

Initial Comment:
Good morning,
I configured postfix to send missed spam mail to dspam through pipe with the following entry in the master.cf configuration file:

spam   unix    -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/local/bin/dspamc --user root --class=spam --source=error

It seems that the system works fine but in the maillog i have the following error:

Feb 18 16:06:54 dspam dspam[19530]: Signature retrieval for '122,512239e722133388817095' failed
Feb 18 16:06:54 dspam dspam[19530]: Unable to find a valid signature. Aborting.
Feb 18 16:06:54 dspam dspam[19530]: process_message returned error -5.  dropping message.

If I make an sql query on DB, i can find a record with signature = '122,512239e722133388817095'

what was wrong?

Thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605201&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3605197 ] problem with email retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3605197, was opened at 2013-02-18 07:42
Message generated for change (Tracker Item Submitted) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605197&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Nobody/Anonymous (nobody)
Summary: problem with email retraining

Initial Comment:
Good morning,
I configured postfix to send missed spam mail to dspam through pipe with the following entry in the master.cf configuration file:

spam   unix    -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/local/bin/dspamc --user root --class=spam --source=error

It seems that the system works fine but in the maillog i have the following error:

Feb 18 16:06:54 dspam dspam[19530]: Signature retrieval for '122,512239e722133388817095' failed
Feb 18 16:06:54 dspam dspam[19530]: Unable to find a valid signature. Aborting.
Feb 18 16:06:54 dspam dspam[19530]: process_message returned error -5.  dropping message.

If I make an sql query on DB, i can find a record with signature = '122,512239e722133388817095'

what was wrong?

Thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3605197&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Comment added) made by bcully
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
Assigned to: Stevan Bajic (sbajic)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

Comment By: Brendan Cully (bcully)
Date: 2013-02-01 14:28

Message:
Renaming the headers is probably a good idea (I guess dspam will still
treat them as spam/notspam signals the same way even if they're under a
different name), but I think it would still be a good idea for dspam to
search the entire header set for the signature when retraining.

----------------------------------------------------------------------

Comment By: Tom Hendrikx (tomhendr)
Date: 2013-02-01 13:36

Message:
Typically you would configure your MTA to remove or rename the untrusted
headers before the message reaches DSPAM. It depends on the way you built
your mail stack, but I use this in postfix:

main.cf:
header_checks = pcre:/etc/postfix/header_checks.pcre

header_checks.pcre:
# Rename headers of incoming messages that might interfere with our own
setup.
/^X-(DSPAM-.*): (.*)$/m REPLACE X-External-$1: $2


----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 13:25

Message:
Thank you for clarification. Now I understand.

----------------------------------------------------------------------

Comment By: Brendan Cully (bcully)
Date: 2013-02-01 13:15

Message:
I received a message that already had a set of DSPAM headers (generated to
fool spam filters, presumably). When my dspam received it, it misclassified
it as innocent and added its own second set of DSPAM headers:

From: ...
Subject: ...
Received...
X-DSPAM-Signature: bogussig
...
Received...
X-DSPAM-Result: Innocent
X-DSPAM-Signature: goodsig

The first set of DSPAM headers are fake, created as part of the spam
message to fool spam detection. The second set are generated by my DSPAM
instance when I receive the message.

I tried to reclassify the message as spam by bouncing it to my
spam-foo@domain address, which pipes it trhough dspamc --class=spam
--source=error. But reclassification failed because it took the signature
from the first X-DSPAM-Signature header, couldn't find it in my user
database, and gave up. If it had continued looking for the signature
instead of giving up, it would have found the second, valid signature and
succeeded at reclassifying the message.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 12:32

Message:
Hello Brendan, I don't understand what the problem is? Can you please
rephrase your problem? How did you pass that message to dspamc for
retraining?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Comment added) made by tomhendr
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
Assigned to: Stevan Bajic (sbajic)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

>Comment By: Tom Hendrikx (tomhendr)
Date: 2013-02-01 13:36

Message:
Typically you would configure your MTA to remove or rename the untrusted
headers before the message reaches DSPAM. It depends on the way you built
your mail stack, but I use this in postfix:

main.cf:
header_checks = pcre:/etc/postfix/header_checks.pcre

header_checks.pcre:
# Rename headers of incoming messages that might interfere with our own
setup.
/^X-(DSPAM-.*): (.*)$/m REPLACE X-External-$1: $2


----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 13:25

Message:
Thank you for clarification. Now I understand.

----------------------------------------------------------------------

Comment By: Brendan Cully (bcully)
Date: 2013-02-01 13:15

Message:
I received a message that already had a set of DSPAM headers (generated to
fool spam filters, presumably). When my dspam received it, it misclassified
it as innocent and added its own second set of DSPAM headers:

From: ...
Subject: ...
Received...
X-DSPAM-Signature: bogussig
...
Received...
X-DSPAM-Result: Innocent
X-DSPAM-Signature: goodsig

The first set of DSPAM headers are fake, created as part of the spam
message to fool spam detection. The second set are generated by my DSPAM
instance when I receive the message.

I tried to reclassify the message as spam by bouncing it to my
spam-foo@domain address, which pipes it trhough dspamc --class=spam
--source=error. But reclassification failed because it took the signature
from the first X-DSPAM-Signature header, couldn't find it in my user
database, and gave up. If it had continued looking for the signature
instead of giving up, it would have found the second, valid signature and
succeeded at reclassifying the message.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 12:32

Message:
Hello Brendan, I don't understand what the problem is? Can you please
rephrase your problem? How did you pass that message to dspamc for
retraining?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Comment added) made by sbajic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
Assigned to: Stevan Bajic (sbajic)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

>Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 13:25

Message:
Thank you for clarification. Now I understand.

----------------------------------------------------------------------

Comment By: Brendan Cully (bcully)
Date: 2013-02-01 13:15

Message:
I received a message that already had a set of DSPAM headers (generated to
fool spam filters, presumably). When my dspam received it, it misclassified
it as innocent and added its own second set of DSPAM headers:

From: ...
Subject: ...
Received...
X-DSPAM-Signature: bogussig
...
Received...
X-DSPAM-Result: Innocent
X-DSPAM-Signature: goodsig

The first set of DSPAM headers are fake, created as part of the spam
message to fool spam detection. The second set are generated by my DSPAM
instance when I receive the message.

I tried to reclassify the message as spam by bouncing it to my
spam-foo@domain address, which pipes it trhough dspamc --class=spam
--source=error. But reclassification failed because it took the signature
from the first X-DSPAM-Signature header, couldn't find it in my user
database, and gave up. If it had continued looking for the signature
instead of giving up, it would have found the second, valid signature and
succeeded at reclassifying the message.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 12:32

Message:
Hello Brendan, I don't understand what the problem is? Can you please
rephrase your problem? How did you pass that message to dspamc for
retraining?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Comment added) made by bcully
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
Assigned to: Stevan Bajic (sbajic)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

Comment By: Brendan Cully (bcully)
Date: 2013-02-01 13:15

Message:
I received a message that already had a set of DSPAM headers (generated to
fool spam filters, presumably). When my dspam received it, it misclassified
it as innocent and added its own second set of DSPAM headers:

From: ...
Subject: ...
Received...
X-DSPAM-Signature: bogussig
...
Received...
X-DSPAM-Result: Innocent
X-DSPAM-Signature: goodsig

The first set of DSPAM headers are fake, created as part of the spam
message to fool spam detection. The second set are generated by my DSPAM
instance when I receive the message.

I tried to reclassify the message as spam by bouncing it to my
spam-foo@domain address, which pipes it trhough dspamc --class=spam
--source=error. But reclassification failed because it took the signature
from the first X-DSPAM-Signature header, couldn't find it in my user
database, and gave up. If it had continued looking for the signature
instead of giving up, it would have found the second, valid signature and
succeeded at reclassifying the message.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 12:32

Message:
Hello Brendan, I don't understand what the problem is? Can you please
rephrase your problem? How did you pass that message to dspamc for
retraining?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Comment added) made by sbajic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
>Assigned to: Stevan Bajic (sbajic)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

>Comment By: Stevan Bajic (sbajic)
Date: 2013-02-01 12:32

Message:
Hello Brendan, I don't understand what the problem is? Can you please
rephrase your problem? How did you pass that message to dspamc for
retraining?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3603022 ] False DSPAM headers confuse dspamc retraining

[SourceForge.net <no...@so...>]

Bug Tracker item #3603022, was opened at 2013-02-01 12:22
Message generated for change (Tracker Item Submitted) made by bcully
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brendan Cully (bcully)
Assigned to: Nobody/Anonymous (nobody)
Summary: False DSPAM headers confuse dspamc retraining

Initial Comment:
I just received a spam email which included its own DSPAM headers (imitation is the sincerest form of flattery!), including:
X-DSPAM-Signature: 1,510c1e2526783905217225

The message also had legitimate DSPAM headers added by my dspam at the very end of the headers, but when I sent the message to dspamc for retraining, it errored out at the bogus header:
Feb  1 12:07:41 xxx dspam[xxx]: Signature retrieval for '1,510c1e2526783905217225' failed
Feb  1 12:07:41 xxx dspam[xxx]: Unable to find a valid signature. Aborting.

Perhaps dspam should keep looking if lookup fails, instead of giving up immediately?

Walking the headers backwards would also work for me since dspam's are at the end, but I could see that breaking with more complicated email flows that might involve passing through more than one dspam instance.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3603022&group_id=250683

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Benny P. <me...@ju...>]

Jani Partanen skrev den 23-01-2013 19:28:
> I would like to toss in this mess again my idea what I already 
> mention
> few years ago here, but don't remember end result.

+1

> Would it be time to add support for db quarantine?

so its time to consider when mysql/postgresql/sqlite3 can handle it ?

> Dspam already use
> database, so it should not be so hard to store quarantine messages 
> into
> database.

correct, but filesystem is already a database, no ?

> It would make Dspam much more scalable, example together with dbmail 
> you
> could do all kind of magic with db replication etc...

super to restore emails if postgresql is out with a new version :|&>

i still like to dream

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Tod A. S. <san...@ri...>]

On 8.1.2013 4:33, Troy Ayers wrote:
> Could I trouble you for a sanitized version of your /usr/site/dspam_train/bin/dspam_fp script?  My system is ...

I am so sorry.  I missed your original email a couple weeks ago.  I will attach the script we use.

The bad news is that the script relies on a slightly patched DSPAM: it causes DSPAM to print error messages to STDERR in addition to it's normal logging.   The patch is only a few lines, and we've used it for many years.  However, I don't know if it's OK for the general community.  I'll attach that patch too.

Before invoking DSPAM for retraining, you could verify that the signature exists; if it does not, don't invoke DSPAM - invoke direct delivery instead.  We implimented this for testing long ago, but we decided on the STDERR route.



Tod Sandman
Sr. Systems Administrator
Middleware Development & Integration
Rice University
Voice: 713.348.5816

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Paul C. <ds...@cy...>]

I've had discussions about using IMAP for storage, but no developer resources seem keen to make it happen.




Jani Partanen <ji...@so...> wrote:

>I would like to toss in this mess again my idea what I already mention 
>few years ago here, but don't remember end result.
>
>Would it be time to add support for db quarantine? Dspam already use 
>database, so it should not be so hard to store quarantine messages into
>
>database.
>It would make Dspam much more scalable, example together with dbmail
>you 
>could do all kind of magic with db replication etc...
>
>// JiiPee
>
>
>On 8.1.2013 4:33, Troy Ayers wrote:
>> Could I trouble you for a sanitized version of your
>> /usr/site/dspam_train/bin/dspam_fp script?  My system is
>> Linux/sendmail/procmail.   Hate to re-invent the wheel... I'm having
>> trouble envisioning how I would verify success or failure of delivery
>by
>> DSPAM.  Does it involve checking the users retrain.log?
>>
>> (thinking out loud) I should check for a mailbox full condition too
>> (Quarantine mbox is separate from per user quotas in my case.)
>>
>> -Troy
>>
>>
>>
>> On 1/7/2013 7:24 PM, Tod A. Sandman wrote:
>>> Here is another patch for dspam.cgi that deals with the delivery of
>false positives from the quarantine.  It is a patch on top of the
>recent quarantine file locking patch I submitted.  It allows for
>specifying in configure.pl an external command for delivering false
>positives.  If this command is not defined, the patch has no effect,
>other than some optional additional logging.
>>>
>>> For us, this is a script that first attempts to retrain/deliver with
>dspam as normal.  But then the script tries to verify if the message
>was actually delivered, and if not, it will try to deliver the command
>directly (for instance, with procmail).  This external script is quite
>site specific.
>>>
>>>
>>> Background:
>>>
>>> If the DSPAM signature of a message cannot be found in the database
>(most often because it has expired and has been cleaned from the
>database), the dspam command will exit with a successful status even
>though it does not deliver the message.
>>>
>>>
>>> Long ago, we had several unhappy professors after they returned from
>sabbatical and attempted to deliver old messages in their quarantines.
>>>
>>>
>>>
>>> Tod Sandman
>>> Sr. Systems Administrator
>>> Middleware Development & Integration
>>> Rice University
>>>
>>>
>>>
>------------------------------------------------------------------------------
>>> Master SQL Server Development, Administration, T-SQL, SSAS, SSIS,
>SSRS
>>> and more. Get SQL Server skills now (including 2012) with
>LearnDevNow -
>>> 200+ hours of step-by-step video tutorials by Microsoft MVPs and
>experts.
>>> SALE $99.99 this month only - learn more at:
>>> http://p.sf.net/sfu/learnmore_122512
>>>
>>>
>>> _______________________________________________
>>> Dspam-devel mailing list
>>> Dsp...@li...
>>> https://lists.sourceforge.net/lists/listinfo/dspam-devel
>>
>>
>>
>>
>------------------------------------------------------------------------------
>> Master SQL Server Development, Administration, T-SQL, SSAS, SSIS,
>SSRS
>> and more. Get SQL Server skills now (including 2012) with LearnDevNow
>-
>> 200+ hours of step-by-step video tutorials by Microsoft MVPs and
>experts.
>> SALE $99.99 this month only - learn more at:
>> http://p.sf.net/sfu/learnmore_122512
>>
>>
>>
>> _______________________________________________
>> Dspam-devel mailing list
>> Dsp...@li...
>> https://lists.sourceforge.net/lists/listinfo/dspam-devel
>>
>
>
>------------------------------------------------------------------------------
>Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
>MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
>with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>MVPs and experts. ON SALE this month only -- learn more at:
>http://p.sf.net/sfu/learnnow-d2d
>_______________________________________________
>Dspam-devel mailing list
>Dsp...@li...
>https://lists.sourceforge.net/lists/listinfo/dspam-devel

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Jani P. <ji...@so...>]

I would like to toss in this mess again my idea what I already mention 
few years ago here, but don't remember end result.

Would it be time to add support for db quarantine? Dspam already use 
database, so it should not be so hard to store quarantine messages into 
database.
It would make Dspam much more scalable, example together with dbmail you 
could do all kind of magic with db replication etc...

// JiiPee


On 8.1.2013 4:33, Troy Ayers wrote:
> Could I trouble you for a sanitized version of your
> /usr/site/dspam_train/bin/dspam_fp script?  My system is
> Linux/sendmail/procmail.   Hate to re-invent the wheel... I'm having
> trouble envisioning how I would verify success or failure of delivery by
> DSPAM.  Does it involve checking the users retrain.log?
>
> (thinking out loud) I should check for a mailbox full condition too
> (Quarantine mbox is separate from per user quotas in my case.)
>
> -Troy
>
>
>
> On 1/7/2013 7:24 PM, Tod A. Sandman wrote:
>> Here is another patch for dspam.cgi that deals with the delivery of false positives from the quarantine.  It is a patch on top of the recent quarantine file locking patch I submitted.  It allows for specifying in configure.pl an external command for delivering false positives.  If this command is not defined, the patch has no effect, other than some optional additional logging.
>>
>> For us, this is a script that first attempts to retrain/deliver with dspam as normal.  But then the script tries to verify if the message was actually delivered, and if not, it will try to deliver the command directly (for instance, with procmail).  This external script is quite site specific.
>>
>>
>> Background:
>>
>> If the DSPAM signature of a message cannot be found in the database (most often because it has expired and has been cleaned from the database), the dspam command will exit with a successful status even though it does not deliver the message.
>>
>>
>> Long ago, we had several unhappy professors after they returned from sabbatical and attempted to deliver old messages in their quarantines.
>>
>>
>>
>> Tod Sandman
>> Sr. Systems Administrator
>> Middleware Development & Integration
>> Rice University
>>
>>
>> ------------------------------------------------------------------------------
>> Master SQL Server Development, Administration, T-SQL, SSAS, SSIS, SSRS
>> and more. Get SQL Server skills now (including 2012) with LearnDevNow -
>> 200+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
>> SALE $99.99 this month only - learn more at:
>> http://p.sf.net/sfu/learnmore_122512
>>
>>
>> _______________________________________________
>> Dspam-devel mailing list
>> Dsp...@li...
>> https://lists.sourceforge.net/lists/listinfo/dspam-devel
>
>
>
> ------------------------------------------------------------------------------
> Master SQL Server Development, Administration, T-SQL, SSAS, SSIS, SSRS
> and more. Get SQL Server skills now (including 2012) with LearnDevNow -
> 200+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
> SALE $99.99 this month only - learn more at:
> http://p.sf.net/sfu/learnmore_122512
>
>
>
> _______________________________________________
> Dspam-devel mailing list
> Dsp...@li...
> https://lists.sourceforge.net/lists/listinfo/dspam-devel
>

[Dspam-devel] Why no overflow in dspam.c:503

[Thomas Preud'h. <ro...@ce...>]

Greetings everybody,

I was suggested to incorporate commit [1] in the Debian dspam package. 
However, I was rightfully asked wether this is overflow-safe since strcpy is 
used. I suppose one of the _getline function is used to ensure CTX->username 
always ends with a '\0' and contains at max 1024 bytes but what about size of 
ATX->recipient >= strlen(CTX->username) + 1 ?

[1] 
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=cbed19764df8ecc1469f739ca33eb92aa3f67584

Best regards,

Thomas Preud'homme

[Dspam-devel] [ dspam-Bug Tracker-3600364 ] Problem releasing quarantined email

[SourceForge.net <no...@so...>]

Bug Tracker item #3600364, was opened at 2013-01-11 01:46
Message generated for change (Comment added) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
>Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Stevan Bajic (sbajic)
Summary: Problem releasing quarantined email

Initial Comment:
Hi, I have a strange problem releasing email from quarantine. It seems that dspam "reinject" the released mail with wrong rcpt-to field, so mta discard it.

Ex:

Jan 10 18:10:14 dspam postfix/smtpd[13786]: A771E3D9: client=localhost[127.0.0.1]
Jan 10 18:10:14 dspam postfix/cleanup[13789]: A771E3D9: message-id=<201...@ds...>
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: from=<>, size=801, nrcpt=1 (queue active)
Jan 10 18:10:14 dspam postfix/pipe[13790]: A771E3D9: to=<`?@previnet.it>, orig_to=<`?>, relay=cattura, delay=0.13, delays=0.12/0/0/0, dsn=2.0.0, status=sent (delivered via cattura service)
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: removed
Jan 10 18:10:14 dspam postfix/smtpd[13786]: disconnect from localhost[127.0.0.1]

I'm using dspam in a relay context, so I have an MTA before, than dspam, than another MTA following. I tried to use both Exchange server and postfix as the final MTA, but the problem is still the same. The strange thing is that all is fine if the email is delivered directly. 
I captured (using pipe) the "conversation" from dspam and the last postfix and here the result:

This is from quarantine (piping to dspam)

From QUARANTINE Thu Jan 10 18:07:41 2013
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:41 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178

hi


and this is from dspam:

Return-Path: <MAILER-DAEMON>
Received: from dspam.previnet.it (localhost [127.0.0.1])
        by dspam.previnet.it (Postfix) with SMTP id 57A822F3
        for <ÐÁ>; Fri, 11 Jan 2013 09:56:52 +0100 (CET)
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:42 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178
X-DSPAM-Reclassified: Innocent

hi

Is seems that the message exit corrupted from dspam (ex: for <ÐÁ>). 

Dspam was compiled with the follow paramaters:

./configure --sysconfdir=/data/dspam --with-dspam-home=/data/dspam --enable-domain-scale --with-storage-driver=mysql_drv --enable-daemon --enable-virtual-users --enable-preferences-extension --enable-debug --enable-clamav --with-mysql-libraries=/usr/lib64/mysql --with-mysql-includes=/usr/include/mysql --with-dspam-home-owner=dspam --with-dspam-home-group=root --with-dspam-owner=dspam --with-dspam-group=root

I will finish by saying that i was using with success dspam version 3.8.0, with a similar configuration.

Thanks

----------------------------------------------------------------------

>Comment By: https://www.google.com/accounts ()
Date: 2013-01-11 02:21

Message:
Yessssss! This was the solution. Thank you very much! :-) my
congratulations for your fine work. Have a nice day

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2013-01-11 02:00

Message:
Which version of DSPAM is this exactly?

This patch ->
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=cbed19764df8ecc1469f739ca33eb92aa3f67584
<- should IMHO fix that issue for you. Let me know if not.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3600364 ] Problem releasing quarantined email

[SourceForge.net <no...@so...>]

Bug Tracker item #3600364, was opened at 2013-01-11 01:46
Message generated for change (Comment added) made by sbajic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
>Assigned to: Stevan Bajic (sbajic)
Summary: Problem releasing quarantined email

Initial Comment:
Hi, I have a strange problem releasing email from quarantine. It seems that dspam "reinject" the released mail with wrong rcpt-to field, so mta discard it.

Ex:

Jan 10 18:10:14 dspam postfix/smtpd[13786]: A771E3D9: client=localhost[127.0.0.1]
Jan 10 18:10:14 dspam postfix/cleanup[13789]: A771E3D9: message-id=<201...@ds...>
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: from=<>, size=801, nrcpt=1 (queue active)
Jan 10 18:10:14 dspam postfix/pipe[13790]: A771E3D9: to=<`?@previnet.it>, orig_to=<`?>, relay=cattura, delay=0.13, delays=0.12/0/0/0, dsn=2.0.0, status=sent (delivered via cattura service)
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: removed
Jan 10 18:10:14 dspam postfix/smtpd[13786]: disconnect from localhost[127.0.0.1]

I'm using dspam in a relay context, so I have an MTA before, than dspam, than another MTA following. I tried to use both Exchange server and postfix as the final MTA, but the problem is still the same. The strange thing is that all is fine if the email is delivered directly. 
I captured (using pipe) the "conversation" from dspam and the last postfix and here the result:

This is from quarantine (piping to dspam)

From QUARANTINE Thu Jan 10 18:07:41 2013
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:41 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178

hi


and this is from dspam:

Return-Path: <MAILER-DAEMON>
Received: from dspam.previnet.it (localhost [127.0.0.1])
        by dspam.previnet.it (Postfix) with SMTP id 57A822F3
        for <ÐÁ>; Fri, 11 Jan 2013 09:56:52 +0100 (CET)
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:42 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178
X-DSPAM-Reclassified: Innocent

hi

Is seems that the message exit corrupted from dspam (ex: for <ÐÁ>). 

Dspam was compiled with the follow paramaters:

./configure --sysconfdir=/data/dspam --with-dspam-home=/data/dspam --enable-domain-scale --with-storage-driver=mysql_drv --enable-daemon --enable-virtual-users --enable-preferences-extension --enable-debug --enable-clamav --with-mysql-libraries=/usr/lib64/mysql --with-mysql-includes=/usr/include/mysql --with-dspam-home-owner=dspam --with-dspam-home-group=root --with-dspam-owner=dspam --with-dspam-group=root

I will finish by saying that i was using with success dspam version 3.8.0, with a similar configuration.

Thanks

----------------------------------------------------------------------

>Comment By: Stevan Bajic (sbajic)
Date: 2013-01-11 02:00

Message:
Which version of DSPAM is this exactly?

This patch ->
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=cbed19764df8ecc1469f739ca33eb92aa3f67584
<- should IMHO fix that issue for you. Let me know if not.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

[Dspam-devel] [ dspam-Bug Tracker-3600364 ] Problem releasing quarantined email

[SourceForge.net <no...@so...>]

Bug Tracker item #3600364, was opened at 2013-01-11 01:46
Message generated for change (Tracker Item Submitted) made by 
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.10.x
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: https://www.google.com/accounts ()
Assigned to: Nobody/Anonymous (nobody)
Summary: Problem releasing quarantined email

Initial Comment:
Hi, I have a strange problem releasing email from quarantine. It seems that dspam "reinject" the released mail with wrong rcpt-to field, so mta discard it.

Ex:

Jan 10 18:10:14 dspam postfix/smtpd[13786]: A771E3D9: client=localhost[127.0.0.1]
Jan 10 18:10:14 dspam postfix/cleanup[13789]: A771E3D9: message-id=<201...@ds...>
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: from=<>, size=801, nrcpt=1 (queue active)
Jan 10 18:10:14 dspam postfix/pipe[13790]: A771E3D9: to=<`?@previnet.it>, orig_to=<`?>, relay=cattura, delay=0.13, delays=0.12/0/0/0, dsn=2.0.0, status=sent (delivered via cattura service)
Jan 10 18:10:14 dspam postfix/qmgr[28819]: A771E3D9: removed
Jan 10 18:10:14 dspam postfix/smtpd[13786]: disconnect from localhost[127.0.0.1]

I'm using dspam in a relay context, so I have an MTA before, than dspam, than another MTA following. I tried to use both Exchange server and postfix as the final MTA, but the problem is still the same. The strange thing is that all is fine if the email is delivered directly. 
I captured (using pipe) the "conversation" from dspam and the last postfix and here the result:

This is from quarantine (piping to dspam)

From QUARANTINE Thu Jan 10 18:07:41 2013
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:41 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178

hi


and this is from dspam:

Return-Path: <MAILER-DAEMON>
Received: from dspam.previnet.it (localhost [127.0.0.1])
        by dspam.previnet.it (Postfix) with SMTP id 57A822F3
        for <ÐÁ>; Fri, 11 Jan 2013 09:56:52 +0100 (CET)
Received: by dspam.previnet.it (Postfix, from userid 0)
        id CDCA43DD; Thu, 10 Jan 2013 17:07:42 +0000 (UTC)
Date: Thu, 10 Jan 2013 18:07:41 +0100
To: sis...@pr...
Subject: prova
User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <201...@ds...>
From: ro...@pr... (root)
X-DSPAM-Result: Spam
X-DSPAM-Processed: Thu Jan 10 18:07:41 2013
X-DSPAM-Confidence: 0.6215
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2,50eef55e133377179792178
X-DSPAM-Reclassified: Innocent

hi

Is seems that the message exit corrupted from dspam (ex: for <ÐÁ>). 

Dspam was compiled with the follow paramaters:

./configure --sysconfdir=/data/dspam --with-dspam-home=/data/dspam --enable-domain-scale --with-storage-driver=mysql_drv --enable-daemon --enable-virtual-users --enable-preferences-extension --enable-debug --enable-clamav --with-mysql-libraries=/usr/lib64/mysql --with-mysql-includes=/usr/include/mysql --with-dspam-home-owner=dspam --with-dspam-home-group=root --with-dspam-owner=dspam --with-dspam-group=root

I will finish by saying that i was using with success dspam version 3.8.0, with a similar configuration.

Thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3600364&group_id=250683

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Troy A. <dsp...@wc...>]

Could I trouble you for a sanitized version of your  
/usr/site/dspam_train/bin/dspam_fp script?  My system is 
Linux/sendmail/procmail.   Hate to re-invent the wheel... I'm having 
trouble envisioning how I would verify success or failure of delivery by 
DSPAM.  Does it involve checking the users retrain.log?

(thinking out loud) I should check for a mailbox full condition too 
(Quarantine mbox is separate from per user quotas in my case.)

-Troy



On 1/7/2013 7:24 PM, Tod A. Sandman wrote:
> Here is another patch for dspam.cgi that deals with the delivery of false positives from the quarantine.  It is a patch on top of the recent quarantine file locking patch I submitted.  It allows for specifying in configure.pl an external command for delivering false positives.  If this command is not defined, the patch has no effect, other than some optional additional logging.
>
> For us, this is a script that first attempts to retrain/deliver with dspam as normal.  But then the script tries to verify if the message was actually delivered, and if not, it will try to deliver the command directly (for instance, with procmail).  This external script is quite site specific.
>
>
> Background:
>
> If the DSPAM signature of a message cannot be found in the database (most often because it has expired and has been cleaned from the database), the dspam command will exit with a successful status even though it does not deliver the message.
>
>
> Long ago, we had several unhappy professors after they returned from sabbatical and attempted to deliver old messages in their quarantines.
>
>
>
> Tod Sandman
> Sr. Systems Administrator
> Middleware Development & Integration
> Rice University
>
>
> ------------------------------------------------------------------------------
> Master SQL Server Development, Administration, T-SQL, SSAS, SSIS, SSRS
> and more. Get SQL Server skills now (including 2012) with LearnDevNow -
> 200+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
> SALE $99.99 this month only - learn more at:
> http://p.sf.net/sfu/learnmore_122512
>
>
> _______________________________________________
> Dspam-devel mailing list
> Dsp...@li...
> https://lists.sourceforge.net/lists/listinfo/dspam-devel

Re: [Dspam-devel] More patches for dspam.cgi: delivery of false positives from quarantine

[Troy A. <dsp...@wc...>]

indeed!

Another scenario: Customer opts of dspam.  Purge signature & token data 
for the account... But leave the quarantine intact (we have a cron job 
to  purged old messages).  But then you can't recover messages from the 
quarantine with the web gui, as you will encounter the behavior below.  
Not unless you can restore signature and token data...  Or manipulate 
the mbox quarantine file directly.  If the customer hasn't tried to 
retrain the message already.  It's a mess.

Dspam should never lose email due to an unexpected DB result.

Thanks a ton Tod!  I'm excited to try this out.

-Troy

On 1/7/2013 7:24 PM, Tod A. Sandman wrote:
> If the DSPAM signature of a message cannot be found in the database (most often because it has expired and has been cleaned from the database), the dspam command will exit with a successful status even though it does not deliver the message.