[Dspam-devel] [ dspam-Bug Tracker-2981066 ] email-addr upper|lower case

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bug Tracker item #2981066, was opened at 2010-04-02 13:05
Message generated for change (Comment added) made by sbajic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2981066&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.9.0
Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By:  LS999 (ls999)
Assigned to: Stevan Bajic (sbajic)
Summary: email-addr upper|lower case 

Initial Comment:
I'm use now hash_drv for testing dspam & found next problem: 'To' & 'From' mail fields can use uppper & lower case symbols (it not impotant for smtp). As result DB can contain some info for one email (Aa@MyDomain.No, Aa...@my... and more more, more...). 

Result: 
- DB contain all versions of AA@MYDOMAIN.NO in all variantes in five days in my case  (Oo-disk space is not rubber!)
- if mail send to Aa@MyEmail.NO i can't retrain DB when send message later from aa...@my... to sp...@my.... will see:

"Apr  2 13:37:37 main dspam[26224]: Unable to open file for reading: /var/local/dspam/data/mydomain.no/aa/aa.sig/4bb554fb268827715621127.sig: No   such file or directory
Apr  2 13:37:37 main dspam[26224]: Signature retrieval for '4bb554fb268827715621127' failed
Apr  2 13:37:37 main dspam[26224]: Unable to find a valid signature. Aborting.
Apr  2 13:37:37 main dspam[26224]: process_message returned error -5.  dropping message."

but real signature is present in the /var/local/dspam/data/MyEmail.NO/Aa/Aa.sig/ directory

IMHO it some smtp-server depended & realy possibility limits when use another processing what described in README (i realy can rewrite this data using smtp-server abilites, but it will global for all another processes after dspam. and it not bes way for me as example. it MUST work internally)

one-side: all email-addr must processed as is
another-side: subsequent analysis must be case-insensitive

IMHO daemon CAN (&mb must) process incoming email-addr as is, for collect data, but database processing (exactly - signatures find) must be case-insensitive|config options regulated

PS as add: require to future 'opt-in' 'opt-out' domain based checks when compuled with --enable-domain-scale options  (global l level is, user level is, where is domain level?)

PSS Oooo and more: custom queries to DB  in conf. I think not only me already have & use self user+servises databases. Another DB from U & permanent synk on every UID  change not show best (therefore i try hash_drv engine for testing now).

TY & Best regards.

----------------------------------------------------------------------

>Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 03:35

Message:
I don't take it as an offense. I just don't understand. In all your stages
you want your system to: me...@my... != ME@MYDOMAIN.NO

But on the DSPAM storage you want: me...@my... == ME@MYDOMAIN.NO

And this is not logical to me. Do you understand what I mean?

Look. It is ultra giga easy for me to code that lowercasing stuff on the
storage. It is just a bunch of lines in C code. But again: IT IS NOT
LOGICAL.

If your first MTA is Postfix and your content filter is DSPAM and your
delivery host is Microsoft Exchange then you ask me to:
* Postfix should threat the recipient my...@do... as a different user
then MY@DOMAIN.NO
* DSPAM should threat my...@do... THE SAME as MY@DOMAIN.NO but ONLY
INTERNALLY. You don't want DSPAM to change headers nor anything other.
Everything should stay as it is but internally it should make that
lowercasing stuff BUT NOT SHOW IT TO THE OUTSIDE that it did that.
* Microsoft Exchange then again should threat my...@do... DIFFERENT then
MY@DOMAIN.NO

Do you understand that this is wicked? Do you understand that DSPAM would
then merge two absolutely different accounts into one account? Do you see
the problem I am seeing and the issue I have with it?

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:28

Message:
Do not take offence.

&Reread self plz.. It child.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 03:26

Message:
> because DSPAM IS NOT FINAL point for detecting spam etc. It ONLY 1step
in
> chain. Therefore it can't invalidate incoming information. (now i'm not
> native fully)
> 
I understand this. But why do you want DSPAM to use ONE FREAKING storage
for my...@do... and MY@DOMAIN.NO and MY...@do... while on every other
stage in your decision you make a difference between My@Domain.no,
MY@DOMAIN.NO, my@DoMain.NO, etc?

This is NOT LOGICAL to me.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 03:24

Message:
> ClamAV same filter. why it not change nothing if i want. simple add
> X-header. & final processing i can make as I want.
>
BECAUSE ClamAV does not save any signature or other things on the file
system. THAT is different then what DSPAM does. DSPAM is saving a
signature/message on the filesystem because it needs that information for
retraining. If you don't want DSPAM to save that information on the local
disc then just run DSPAM in "NOTRAIN" mode. When doing this then you can
compare ClamAV with DSPAM.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:24

Message:
"
don't get it. Why should a mail-filter/content-filter act differently on
mail headers then a normal mail client or mail server?
"
because DSPAM IS NOT FINAL point for detecting spam etc. It ONLY 1step in
chain. Therefore it can't invalidate incoming information. (now i'm not
native fully)

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 03:22

Message:
I really have a hard time understanding your English.

> but now any another filter flowed DSPAM will receive "a"? but not "A".
> what about this?
>
Then DO NOT USE "Broken case". That easy!

You totally confuse me. First you DON'T WANT any case to be converted in
lowercase by DSPAM. Perfect. Then don't change anything inside DSPAM and
DSPAM will NOT change anything.

Then you bark that if the case is not changed to lowercase then you end up
in having:
/var/local/dspam/data/mydomain.no/aa/aa.sig/
/var/local/dspam/data/MYDOMEIN.NO/AA/AA.sig/
/var/local/dspam/data/MyDomein.No/aa/aa.sig/
etc...

So then I told you to use "Broken case" but that does not satisfy you
because you still want on the filesystem the name to be converted to
lowercase but not on the header. Right?

So all you want is a case transformation on the file system BUT NOT in the
header. RIGHT?

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:17

Message:
ClamAV same filter. why it not change nothing if i want. simple add
X-header. & final processing i can make as I want.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:15

Message:
fine: 
"
You mean for the content inspection? Definitively not! When DSPAM is
inspecting the content of a message then "A" is not the same as "a".
"
but now any another filter flowed DSPAM  will receive "a"? but not "A".
what about this?

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 03:12

Message:
> Right - FOR MAIL CLIENT & SERVERS ONLY. BUT NOT FOR
> MAIL-FILTER services.
> 
I don't get it. Why should a mail-filter/content-filter act differently on
mail headers then a normal mail client or mail server?

> sayb me one - what algorithms used dsapm.
> 
In what context? I don't understand this question.

> say me second - "a" and "A" for this algorithms is same?
> 
You mean for the content inspection? Definitively not! When DSPAM is
inspecting the content of a message then "A" is not the same as "a".

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:11

Message:
Simple not sleep. What about RFC? Fine - this project not make final mail
receiver - it filter only. And therefore it MUST have ability for don't 
change ANY INCOMING INFORMATION.

i Repeate:
"
decision on mail processing MUST be dspam independent and data
falsification inadmissible here.
"

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 03:02

Message:
"Now according to the RFC the domain part can be in any case. Right? So
DOMAIN.NO is the same as domain.no and is the same as DoMaIn.No, etc...
Right?"
Right - FOR MAIL CLIENT & SERVERS ONLY.  BUT NOT FOR MAIL-FILTER services.
sayb me one  - what algorithms used dsapm. & say me second - "a" and "A"
for this algorithms is same?

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-31 02:07

Message:
My point is:

Message from the internet is arriving for the recipient MY@DOMAIN.NO.

Now according to the RFC the domain part can be in any case. Right? So
DOMAIN.NO is the same as domain.no and is the same as DoMaIn.No, etc...
Right?

The local part is case sensitive according to the RFC's. Right? So
my@DOMAIN.NO is not the same as MY@DOMAIN.NO. Right?

I then told you to look at "Broken case". And you replied that it changes
the header. Right?

Okay. Point taken. It does change the header. So a message send to
MY@DOMAIN.NO will be changed to my...@do... if one uses "Broken case" and
then when DSPAM delivers that message to your delivery host then the
recipient will be my...@do... but you don't want that. You want it to be
MY@DOMAIN.NO while in DSPAM you want it to be my...@do... because you want
to save the message in the DSPAM storage in lowercase. Right?

And here comes my problem. Your MTA should see the message as
"MY@DOMAIN.NO", while DSPAM should see it as "my...@do...", while yet
again your delivery host should see it as "MY@DOMAIN.NO".

Or to say it in other words:
Your MTA should see "MY@DOMAIN.NO" and "my...@do..." as TWO different
accounts, while DSPAM should use ONE single storage/account for both
addresses and your delivery host should then again see TWO different
accounts.

Do you spot the conflict I am having? From a programming viewpoint it is
not difficult to do what you want but I really fail to see the logic behind
it. Either one is consistent or not. And what you request has no logic to
me. If I look at this scenario:

(A) INTERNET --> (B) MTA --> (C) DSPAM --> (D) MTA/DELIVERY

Then I fail to see why A + B + D should be case sensitive while C should
not be case sensitive. What is the logic behind this? Explain this to me.
Please!

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 00:45

Message:
1Q & 1A must be exchanged. Sorry.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-31 00:12

Message:
2sbajic

Sory - very many hard work now... I reread U message about

"
Anyway... if I understand you right, then all you want is that DSPAM does
NOT change any original 'To' headers but use one single store for
signature
writing/reading. Right?
"
i think it what I want, but i will test (as I say in prev message i real
long time don't track project & as U say "it not my native language").

if I understand U righ then now I can use CLI dsapm_admin to resolve
described problems.

1Q: dspam-server has now options for this functionality or I must
permanent use dspam_admin? 
1A: = aswer to U Q 
"
am now going to close this bug report since DSPAM already offers out of
the box the functionality you asked for.
"

TY for work.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-30 23:35

Message:
I think it about item 2. I'm try using another words.

If i have thousands users (my...@do..., my...@do..., ...,
my...@do..., ..., my...@do...) then i must perform (as you say)
for everyone:
     # dspam_admin change preference MY@DOMAIN.NO localStore my...@do...
     # dspam_admin change preference my...@do... localStore my...@do...
It manually or script's work (don't forget that I can't use a web-if). 

This way is impossible in my case:
1) manually processing is simple delirium  for thousands users;
2) script processing is possible as fact, but realtime reaction to changes
in mail system (add new e-mail client for example) is impossible (cron) or
very mail system depended. Mail system here not ONLY smtp-server, but ALL
mail system components & services & it configuration. => many
CURRENT-mail-system depended scripts.
2a) this realization type is time critical when any crash of mail system
is occured (because it very system depended & stupid support can't real
make anything => call me... call me...call me... My...@do...
...MyN+?An...@do... -  i want sleep  sometimes :).

Therefore i say again - this is dspam-server problem:
1) if dspam CAN act as simple filter for add marks then it CAN'T & MUST
don't change ANY another info in incoming mail on self output (it deforms
the information received by following filters) in this mode. The final
decision on mail processing MUST be dspam independent and data
falsification inadmissible here.
1a) this mode must be configurable by server options (not all need such
functioning and i understand it)

PS I don't track this project long time (considering speed of
development). Mb it all already resolved. While I am compelled to use other
decision, but I don't leave dspam ). 
TY 4 all & Personal 4 U sbajic.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-29 10:42

Message:
> 2. what about MY1,My2,My...,MyN? manual processing? 
>
I don't understand that. Can you rephrase that question?

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-05-29 03:44

Message:
Your are right 100%. Not my native language.

1. i'm don't use web (& can't)
2. what about MY1,My2,My...,MyN? manual processing? 

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-05-13 20:49

Message:
I have real hard time to understand your writing. Probably English is not
your native language (it is not my native language too)?

Anyway... if I understand you right, then all you want is that DSPAM does
NOT change any original 'To' headers but use one single store for signature
writing/reading. Right?

Well... DSPAM has already that functionality. Assume you have the email
MY@DOMAIN.NO and want it to be threated the same way as my...@do... then
just do this:
- DO NOT enable "Broken case"
- use dspam_admin to change the store for both emails:
  # dspam_admin change preference MY@DOMAIN.NO localStore my...@do...
  # dspam_admin change preference my...@do... localStore my...@do...

That's it. All mails now arriving to "MY@DOMAIN.NO" or "my...@do..." will
use "my...@do..." as their store.

I am now going to close this bug report since DSPAM already offers out of
the box the functionality you asked for.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-04-06 00:39

Message:
I don't think what You are right.
It's very situation depended. DSPAM acts as the filter in most cases for
add X-headers only. Therefore in this situation:

data->[DSPAM]->[filter_1]->[filter_2]->...

already "filter_1" (and all following it) don't have original message data
(addr modified by DSPAM). 

Therefore ma...@ex... and MA...@ex... IS different addresses in
this case (as DSPAM outgoing data). And it is very important for adequate
data processing with following filters.

IMHO must be one more config option for forward original address as is.
Thus if this option OFF, the "broken case" acts as now. When new option is
ON, the "broken case" is used for internal data process only, but we have
the original "To" (and similar) field(s) on DSPAM output.

And i don't see RFC contradiction here, but feature improvement only .

Adequate data processing

Best regards.

----------------------------------------------------------------------

Comment By: Ion-Mihai "IOnut" Tetcu (itetcu)
Date: 2010-04-04 10:40

Message:
Per RFC:
- the right-hand part (domain) is case insensitive, so we could lowercase
it.
- the left-hand part case-sensitiveness is local site policy. Personally I
don't think we should alter anything here, as it's valid (although not
recommended) to have ma...@ex... and MA...@ex... different
addresses.

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-04-04 03:01

Message:
Add.

must be
MY@DOMAIN.NO->[DSPAM my...@do... (can be config option regulated)
DSPAM]->MY@DOMAIN.NO

but not
MY@DOMAIN.NO->[DSPAM]->my@DOMAIN.NO
or
MY@DOMAIN.NO->[DSPAM]->my...@do...

----------------------------------------------------------------------

Comment By:  LS999 (ls999)
Date: 2010-04-03 02:54

Message:
TY 4 "broken case" option - i missed it - &sorry). But "broken case" option
in this case is partial decision of a problem. For example:

- DSAPM off. Here i see all letter cases was perfect keeped:

Return-Path: <my...@ra...>
X-Original-To: MY@DOMAIN.NO
Delivered-To: MY@DOMAIN.NO
Received: from mcgi64.rambler.ru (mcgi64.rambler.ru [81.19.67.198])
     by mx.domain.no (MTA) with ESMTP id C5F6818006E
     for <MY@DOMAIN.NO>; Sat, 3 Apr 2010 02:32:15 +0400 (MSD)
Received: from [78.31.73.87] by mcgi64.rambler.ru with HTTP (mailimap);
Sat, 03 Apr 2010 02:32:14 +0400
From: <my...@ra...>
To: <MY@DOMAIN.NO>
Subject: dspam off

- DSPAM on, "broken case" option off. Now before DSPAM we have
"MY@DOMAIN.NO" recipient adress, but after reinjecting mail from DSPAM back
to the mail server (smpt used) we have already "for <my@DOMAIN.NO>":

Return-Path: <my...@ra...>
X-Original-To: my@DOMAIN.NO
Delivered-To: my@DOMAIN.NO
Received: from localhost (localhost.localdomain [127.0.0.1])
     by mx.domain.no (MTA) with SMTP id CFB1A180076
     for <my@DOMAIN.NO>; Sat, 3 Apr 2010 02:25:43 +0400 (MSD)
Received: from mxa.rambler.ru (mxa.rambler.ru [81.19.66.231])
     by [mx.domain.no] (MTA) with ESMTP id 8F0CA18006E
     for <MY@DOMAIN.NO>; Sat, 3 Apr 2010 02:25:41 +0400 (MSD)
...
From: <my...@ra...>
To: <MY@DOMAIN.NO>
Subject: broken case off

- DSPAM on, "broken case" option on. And now we have "MY@DOMAIN.NO" in &
"my...@do..." out. Predictably:

Return-Path: <my...@ra...>
X-Original-To: my...@do...
Delivered-To: my...@do...
Received: from localhost (localhost.localdomain [127.0.0.1])
     by mx...@do... (MTA) with SMTP id 0E095180076
     for <my...@do...>; Sat, 3 Apr 2010 02:29:30 +0400 (MSD)
Received: from mcgi20.rambler.ru (mcgi20.rambler.ru [81.19.67.53])
     by main.eletek.ru (MTA) with ESMTP id BB71318006E
     for <MY@DOMAIN.NO>; Sat, 3 Apr 2010 02:29:27 +0400 (MSD)
Received: from [78.31.73.87] by mcgi20.rambler.ru with HTTP (mailimap);
Sat, 03 Apr 2010 02:29:27 +0400
From: <my...@ra...>
To: <MY@DOMAIN.NO>
Subject: broken case on

But in ALL cases with DSPAM i don't have ORIGINAL message headers on
output & this restriction for the filters following it.

IMHO DSPAM don't need change any original message headers. 

Best regards.

----------------------------------------------------------------------

Comment By: Stevan Bajic (sbajic)
Date: 2010-04-02 14:32

Message:
Have you tried to use "Broken case"?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2981066&group_id=250683