dspam

Perfect! That totally makes sense, and puts down documented exactly what I
was finding. I can work around the rest neatly now, aliasing usernames,
which is good enough (and, I think, prettier, in the webui)

> Then you NEED to use the FULL email address when you retrain.
> You can't just use the domain name. It will not work.
>
Sorry. It will work now with the latest GIT code but it will print out the
warning in the logs.

> Yeah, that makes sense, though the setup is different. The domain
> name is actually all 4 parts -- aria.hydra.blumenthals.com -- and the
> username is aria@aria.hydra.blumenthals.com and test@ as well.
>
Then you NEED to use the FULL email address when you retrain. You can't
just use the domain name. It will not work.

> Why the warning, then, if it's retraining the message correctly?
>
You have luck. What it does now is something that is beneficial for you
but looking it from the configuration viewpoint it does not do honor the
configuration. You said it should use fall back domains but it does not
fall back to a domain. It uses the full user name since the user name you
supplied over the command line does not have a domain part in it. But
fortunately for you it is exactly what you want. You supplied just the
domain name and wanted DSPAM to use it anyway. Before the change the code
would have left aria.hydra.blumenthals.com as user name BUT would have
assigned NULL as domain name. The processing in DSPAM would have been (the
old code did this):
1) check if FallbackDomains is turned on -> YES (in your case)
2) extract everything after the @ (including @) and use this as the new
user name -> in your case that would have been NULL
3) call processing routines to do the action the user wants us to do
4) CRASH

The reason for the crash is that a bunch of functions are not made to
check for NULL user name. I know, I know... it's not good to not check. The
original author of DSPAM does check most parameters but he did that on a
early stage and then later he switches the user name in case
FallbackDomains/fallbackDomain is active but he did not recheck the user
name. And that is the reason for your failure. But we added now those
checks. Not everywhere but there where they are now should be +/- okay for
most cases. Maybe in DSPAM 4.0 we will rework that and make it better?

> (Or, the greater question is, how much flexibility is there in the
> name of the domain user? -- and does the web UI work properly
> with the @domain users? I get "no history" unless I use the
> domain.com version of the username instead of @domain.com,
> with the same uid)
>
That is another issue. It's the WebUI code that does not handle fall back
domains the proper way. The WebUI code is not aware of fall back domains
and has no code to handle that. I would need to extend the WebUI code to
handle that. The big problem is that we have said that for 3.9.0 we are
aiming to keep the changes in the WebUI to an absolute minimum. It's a
corner case. I don't mean the fall back domain issue but the setup you
have: fall back domains + shared group (or any group. It's not important
that it is a shared group) on the fall back domain.

You know what? I am going now to close this case. But I would like you to
open a new feature request or another bug report and request that you need
user handling in the WebUI to be able to cope with "@domain.tld".

Is this a deal?


--
Kind Regards from Switzerland,

Stevan Bajić

Yeah, that makes sense, though the setup is different. The domain name is
actually all 4 parts -- aria.hydra.blumenthals.com -- and the username is
aria@aria.hydra.blumenthals.com and test@ as well.

I'm using this as a relay, so you've got that part dead right, and I get
how the signature lookup works now -- having it off in the conf file was an
oversight from my playing around trying to get postgresql working before.
Sorry for the red herring.

Why the warning, then, if it's retraining the message correctly? (Or, the
greater question is, how much flexibility is there in the name of the
domain user? -- and does the web UI work properly with the @domain users? I
get "no history" unless I use the domain.com version of the username
instead of @domain.com, with the same uid)

> It also /does/ retrain even so, now, so I could theoretically throw
> this into production. What's it not doing that it should, if it is in
> fact retraining the message?
>
It is retraining the message. Allow me to explain.

At the time the message is coming from the internet the user name that
DSPAM is using when it is CLASSIFYING the message is probably
"aria.hydra@blumenthals.com". Right? If so then the group definition you
have in place (@blumenthals.com:shared:*@blumenthals.com) are telling DSPAM
to process that message under the username "@blumenthals.com". And since
you have enabled "MySQLUIDInSignature on" or "PgSQLUIDInSignature on" the
generated signature is having the UID of the user @blumenthals.com"
included into the signature. Then the mail gets delivered to 127.0.0.1 to
port 10025 (probably your MTA). That system behind 127.0.0.1:10025 sees the
recipient "aria.hydra@blumenthals.com" and looks up the info to where to
deliver and does that (probably it delivers to
"aria.hydra.blumenthals.com"). You then get that message and if you retrain
you don't retrain for "aria.hydra.blumenthals.com" but you retain for the
UID of @blumenthals.com (probably the UID is 4). Then DSPAM gets that info
and retrains signature "4,xxxxxxxxxxxxxx". But DSPAM first looks in
dspam.conf to see if you have enabled "MySQLUIDInSignature" or
"PgSQLUIDInSignature". If it sees that this is enabled and then it does a
lookup to see who is user with the UID 4 and sees that this is user
"@blumenthals.com" and uses this as the user name for the training. It does
not care who is sending the retaining instruction. It only cares that the
UID is a known UID and that the signature passed is in the
dspam_signature_data under the UID 4. It then just does the retaining and
switches the needed tokens. And that's it. And since the user
"@blumenthals.com" is not only a normal user but as well a shared group for
all users having "@blumenthals.com" in their email, the training done
benefits all of them. It has an direct influence on all of them. Do you
understand what I mean?


--
Kind Regards from Switzerland,

Stevan Bajić

> Nov 20 19:56:08 24-75-24-220 dspam[24648]: process_users(): Can
> not fallback to domains for username 'aria.hydra.blumenthals.com'
> without @domain part.
>
Great. Your error report did not let me in peace. I read your report and
read it again and read it again and I just could not find the problem. Then
I started to look deeper into the code of DSPAM and then I saw this
dependency for the @ sign in the user name and then I realized that this is
probably your issue. So changed the code to handle the issue.

> I don't entirely understand what it's saying, but at least it flags it.
> It's implying the username has to be @domain? (Is there any reason this
> has to be the case?)
>
Yes. It is called fallbackDOMAIN. So the domain part is the fall back. If
you have "user@example.com" and you have "FallbackDomains on" in dspam.conf
then "example.com" is the domain to fall back to. The user name NEEDS a
domain part. Without it DSPAM does not know what domain to take.

It does not matter that you have the domain as shared group. So to sum it
up... assuming the domain you want to fall back to is "blumenthals.com" and
you want to use a shared group for the fallback then:

1) You should have this in your group file:
@blumenthals.com:shared:*@blumenthals.com

Separating this at the colon means:
1st entry: the GROUP NAME (in your case equal to the fallback domain WITH
the @)
2nd entry: the TYPE OF THE GROUP (in your case a shared group)
3th entry: the MEMBERS OF THE GROUP (in your case everyone having
@blumenthals.com in his/her username)

2) Ensuring that "blumenthals.com" is a fallback domain you need to enable
"FallbackDomains on" in your dspam.conf and you need to have a user with
the name "@blumenthals.com" (the domain WITH a @ in front) in the
preference table with the preference "fallbackDomain=on"

3) Each user you want to participate in this fallback domain scenario
needs to have @lumenthals.com in his DSPAM user name else the the whole
fallback domain stuff IS NOT going to work.

My patch from yesterday is handling all of this and should the user name
not have a @domain part then now DSPAM is not honoring the "fallbackDomain"
preference nor is it honoring the "FallbackDomains on" setting from
dspam.conf. Basically it just ignores them and continues operating as if
those two settings would have never been turned on (since DSPAM can't fall
back to the domain part of the user name because it can't determine the
domain the user uses).

Under a proper configuration you could go on and have user@blumenthals.com
and process any mail for that user with DSPAM without having opened an
account for that user in DSPAM. And this is still working. Even if you
would run DSPAM in OptIn mode and not have OptIn active for
user@blumenthals.com, it would work. Regardless the fact that you never
opted in the user@blumenthals.com account. That would normally not be
possible. Without fallbackDomains that user would not be processed. But now
with my patch from yesterday things change. If you have made an error with
your configuration then DSPAM will continue as usual and try to use the
full user name as if there have not been any FallbackDomains settings in
dspam.conf or any failbackDomain settings in the preference extension.

And to continue to explain my example from above: If you would have made
an error in the configuration then DSPAM will now log that issue. And if
you would have turned OptIn on (so that every user would need to opt into
DSPAM processing) and for what ever reason you would try to process a user
where DSPAM can not extract the domain part from the user name then DSPAM
will continue without and then see that the user in question is not opted
into DSPAM filtering and then it would not process that message for that
user. With fallback domains turned on it would first try to get the domain
name and look if the domain name is used as a fallback domain and if that
would be the case then process the mail for the user not under his name but
under the domain name he has. And that would then work around the issue of
that the user is not opted into DSPAM filtering (since the domain is opted
into DSPAM filtering but again... the user does not need to be if fallback
domains is active).

Can I now close this bug here?


--
Kind Regards from Switzerland,

Stevan Bajić

It also /does/ retrain even so, now, so I could theoretically throw this
into production. What's it not doing that it should, if it is in fact
retraining the message?

Ah, excellent. I do!

Nov 20 19:56:08 24-75-24-220 dspam[24648]: process_users(): Can not
fallback to domains for username 'aria.hydra.blumenthals.com' without
@domain part.

I don't entirely understand what it's saying, but at least it flags it.
It's implying the username has to be @domain? (Is there any reason this has
to be the case?)

This here is not going to work with fallback domains:
--user aria.hydra.blumenthals.com

I submitted yesterday a bunch of patches dealing with that in the proper
way since I suspected something to happen in your setup and I just wanted
to handle that and print out a message informing the user that he/she is
doing wrong. Here is the submitted code:
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=5183294f11e985b242d3817ae47eef9f1d8e9d2c

Do you get another message if you compile DSPAM again from GIT and re-run
the test?

That's running on the commandline, but the same command that dspam.cgi
executes with system()

> How do you run the CGI's in the WebUI (SuEXEC? Other way?)

Directly as CGI. apache is in dspam.conf's Trust list.

- With what user?

domain.com -- the domain user. It's got an entry for @domain.com with the
same uid in dspam_virtual_uids. If I don't, the web UI won't show the
history for the user.

- With what user/group does the WebUI run?

apache

- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')

-rwx--s--- 1 dspam mail 274K Nov 19 00:06 /usr/bin/dspam
dspam: /usr/bin/dspam /etc/dspam /usr/share/dspam
/usr/share/man/man1/dspam.1.gz

The errors you posted are when executing the commands from the command line
or when executing from within the WebUI?

Can you answer the following questions:
- How do you run the CGI's in the WebUI (SuEXEC? Other way?)
- With what user?
- With what user/group does the WebUI run?
- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')

Pardon. I mean that that fixes the asking the wrong question problem, but
leaves the greater problem -- not being able t train from the webui.

I had that in my dspam.conf, same issue.

I've tried with pgsql (now that the compilation bug is fixed, I got that
working) -- same issue. Looks like it's not in any of the db-specific
code:

dspam --source=error --class=innocent
--signature=3,4b072ebc91645760821076 --user aria.hydra.blumenthals.com

gives me

Nov 20 19:08:05 24-75-24-220 dspam[9489]: Unable to determine the
destination user
Nov 20 19:08:05 24-75-24-220 dspam[9489]: Context initialization failed
Nov 20 19:08:05 24-75-24-220 dspam[9489]: process_message returned error
-2. dropping message.

Please add in your dspam.conf this here:
MySQLUIDInSignature on

Restart the daemon and then try again.

Things are added to the database with uid = 4, signature =
4b06b9f741961536915273, for example, but if I strace dspam when retraining
with --signature=4,4b06b9f741961536915273, I see that dspam makes this
query:

write(3, "b\0\0\0\3SELECT data,length FROM dspam_signature_data WHERE
uid=4 AND signature=\"4,4b06b9f741961536915273\"", 102) = 102

Any idea why it'd ask the wrong question?

[root@spamfilter ~]# dspam --version

DSPAM Anti-Spam Suite 3.9.0-BETA4 (agent/library)

Copyright (c) 2002-2009 DSPAM Project
http://dspam.sourceforge.net.

DSPAM may be copied only under the terms of the GNU General Public
License,
a copy of which can be found with the DSPAM distribution kit.

Configuration parameters: '--host=x86_64-redhat-linux-gnu'
'--build=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/usr/com' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--sysconfdir=/etc/dspam'
'--with-dspam-home=/var/spool/dspam' '--with-logdir=/var/log/dspam'
'--enable-shared' '--enable-static' '--enable-daemon'
'--enable-external-lookup' '--enable-clamav' '--enable-debug'
'--enable-syslog' '--enable-long-usernames' '--enable-domain-scale'
'--enable-virtual-users'
'--with-storage-driver=hash_drv,mysql_drv,pgsql_drv'
'--with-mysql-includes=/usr/include/mysql'
'--with-mysql-libraries=/usr/lib64/mysql' 'CFLAGS=-O2 -g -m64
-mtune=generic' 'CXXFLAGS=-O2 -g -m64 -mtune=generic' 'FFLAGS=-O2 -g -m64
-mtune=generic' 'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux'

dspam.conf:

Home /var/spool/dspam
StorageDriver /usr/lib64/dspam/libmysql_drv.so
DeliveryHost 127.0.0.1
DeliveryPort 10025
DeliveryIdent localhost
DeliveryProto SMTP
FallbackDomains on
OnFail error
Trust root
Trust dspam
Trust apache
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
Debug *
DebugOpt process spam fp classify
ClassAliasSpam spam
TrainingMode teft
TestConditionalTraining on
Feature whitelist
Algorithm graham burton
Tokenizer chain
PValue bcr
WebStats on
ImprobabilityDrive on
Preference "spamAction=quarantine"
Preference "signatureLocation=message"
Preference "showFactors=on"
Preference "fallbackDomain=on"
AllowOverride trainingMode
AllowOverride spamAction spamSubject
AllowOverride statisticalSedation
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride signatureLocation
AllowOverride showFactors
AllowOverride optIn optOut
AllowOverride whitelistThreshold
AllowOverride dailyQuarantineSummary
AllowOverride fallbackDomain
MySQLServer /var/lib/mysql/mysql.sock
MySQLUser dspam
MySQLPass redacted
MySQLDb dspam
MySQLReconnect true
MySQLConnectionCache 10
PgSQLServer 127.0.0.1
PgSQLPort 5432
PgSQLUser dspam
PgSQLDb dspam
PgSQLConnectionCache 3
PgSQLUIDInSignature on
HashRecMax 98317
HashAutoExtend on
HashMaxExtents 0
HashExtentSize 49157
HashPctIncrease 10
HashMaxSeek 10
HashConnectionCache 10
Notifications on
PurgeSignatures 14
PurgeNeutral 90
PurgeUnused 90
PurgeHapaxes 30
PurgeHits1S 15
PurgeHits1I 15
LocalMX 127.0.0.1
SystemLog on
UserLog on
Opt out
ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full
ClamAVPort 3310
ClamAVHost 127.0.0.1
ClamAVResponse reject
ServerDomainSocketPath /var/run/dspam/dspam.sock
ServerMode auto
ServerQueueSize 32
ServerMode auto
ServerPass.Relay1 redacted
ServerParameters "--deliver=innocent -d %u"
ServerIdent "spamfilter.blumenthals.com"
ClientHost /var/run/dspam/dspam.sock
ClientIdent redacted@Relay1
ProcessorURLContext on
ProcessorBias on
StripRcptDomain off

Well, it logs and stops the segfault with that patch. Now to determine why
it's null ...

Looking at it now. Still happens with current git, but I'm trying other
usernames in the group to see if that's the problem.

Hallo Aredridel,

do you have any update on this case?

--
Kind Regards from Switzerland,

Stevan Bajić

Hallo Aredridel,

when looking at your stack trace I see that ctx_init() get's called with
an NULL username. This should NOT happen. I could trap that case in
ctx_init(). Better: I must trap that case. Can you apply this patch to the
latest GIT source and try if the crash still presists:
------------------------
diff --git a/src/dspam.c b/src/dspam.c
index 7bfe301..b97f4a5 100644
--- a/src/dspam.c
+++ b/src/dspam.c
@@ -1,4 +1,4 @@
-/* $Id: dspam.c,v 1.378 2009/11/13 02:26:19 sbajic Exp $ */
+/* $Id: dspam.c,v 1.379 2009/11/13 12:44:59 sbajic Exp $ */

/*
DSPAM
@@ -2367,6 +2367,12 @@ NEXT:
*/

DSPAM_CTX *ctx_init(AGENT_CTX *ATX, const char *username) {
+
+ if (username == NULL) {
+ LOG (LOG_CRIT, ERR_AGENT_USER_UNDEFINED);
+ return NULL;
+ }
+
DSPAM_CTX *CTX;
char filename[MAX_FILENAME_LENGTH];
char ctx_group[128] = { 0 };
------------------------

I would say that the crash is gone but you will have a run time error
because DSPAM can not work on a undefined user name.


Kind Regards from Switzerland,

Stevan Bajić

Hallo Aredridel,

I can not reproduce this here. What I have done so far:

dspam.conf:
FallbackDomains on

MySQL:
insert into dspam_virtual_uids (uid,username) values
(null,'@domain.com');

Shell (I run a DSPAM in OptIn mode):
dspam_admin change preference @domain.com fallbackDomain on
dspam_admin change preference @domain.com optOut off
dspam_admin change preference @domain.com optIn on

After that my data in MySQL looks like this:
dspam_virtual_uids:
+------+-------------+
| uid | username |
+------+-------------+
| 2087 | @domain.com |
+------+-------------+

dspam_preferences:
+------+----------------+-------+
| uid | preference | value |
+------+----------------+-------+
| 2087 | fallbackDomain | on |
| 2087 | optIn | on |
| 2087 | optOut | off |
+------+----------------+-------+

group file:
@domain.com:shared:*@domain.com

Forced a learning for a bunch of messages. All of them resulted in the
same output:
nyx ~ # dspam --user @domain.com --deliver=summary --stdout
--source=corpus --class=spam < path/to/message1.txt
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=1.00; signature=N/A
nyx ~ #

Tokens where added to the MySQL table:
> select count(*) from dspam_token_data where uid=2087;
+----------+
| count(*) |
+----------+
| 36377 |
+----------+

Now did a test with a user that does NOT exist in dspam_virtual_uids:
nyx ~ # dspam --user test@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=0.99; signature=2087,4afd3dd512221635636504
X-DSPAM-Result: Spam
X-DSPAM-Processed: Fri Nov 13 12:07:01 2009
X-DSPAM-Confidence: 0.9899
X-DSPAM-Improbability: 1 in 9809 chance of being ham
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2087,4afd3dd512221635636504
nyx ~ #

Looks okay to me. The uid used for that user is the one from the user
"@domain.com". Okay. Now let's try to retrain that message:
nyx ~ # dspam --user test@domain.com --stdout --source=error
--class=innocent --signature=2087,4afd3dd512221635636504
nyx ~ #

Works. I see in the Web-UI that the message got changed. Now retrain it
back (undo) to spam:
nyx ~ # dspam --user test@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3dd512221635636504
nyx ~ #

Works as well.

The only difference I see in your and my setup is that I used "@domain" as
the group name for the shared group. You used "domain.com" but "domain.com"
does not exist as user in the dspam_virtual_uids table so I used
"@domain.com". Anyway... going back to just use "domain.com" for the group
(@domain.com:shared:*@domain.com) lead in the same result. Except that this
time the result from DSPAM was innocent instead of spam but that is
probably because the used user "domain.com" for the shared group does not
exist (only @domain.com exists as an user):
nyx ~ # dspam --user test@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Innocent"; class="Innocent";
probability=0.0000; confidence=0.70; signature=2087,4afd3f6b15271305324895
nyx ~ #

==> The strange thing here is that I get a summary result but "--stdout"
should FORCE the whole message to be displayed to the console (incl. a
summary). Something is strange here. I can however force deliver to show up
on screen by instructing DSPAM to deliver innocent or spam mails (which
should have been selected by the "--stdout" switch already):
nyx ~ # dspam --user test@domain.com --deliver=innocent,spam --stdout
--process < path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: Innocent
X-DSPAM-Processed: Fri Nov 13 12:22:48 2009
X-DSPAM-Confidence: 0.6701
X-DSPAM-Improbability: 1 in 204 chance of being spam
X-DSPAM-Probability: 0.0000
X-DSPAM-Signature: 2087,4afd418820052092820584
nyx ~ #

--> BINGO! That is the right output. It should have been that output from
the beginning. Anyway... this is another issue I need to look after later.

I went ahead and retrained that message to be spam:
nyx ~ # dspam --user test@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3f6b15271305324895
nyx ~ #

The Web-UI changed and the message was switched to spam.


What have I done on my setup that you have not? Can you attach your
dspam.conf and post the output of "dspam --version"?


Kind Regards from Switzerland,

Stevan Bajić

I do have the rest as you suggested.

I have:

domain.com:shared:*@domain.com

The docs were a little unclear how it all fits together, the username
munging, etc.

Hallo Aredridel,

how can you mirror that setup on my end? What do I need to have in the
group file and what in the preference extension?

If I have understood your setup then it's +/- like this:

group file:
@domain.com:shared,managed:*@another.com

preferences for domain.com:
fallbackDomain=on

dspam.conf:
FallbackDomains on


Right?



Kind Regards from Switzerland,

Stevan Bajic