#149 dot stuffing bug

[polaris1]

I tried this in both 3.9.0 and in 3.10.1 with similar results.

I run dspam in client/daemon mode ( daemon is 3.10.1, clients I tried both 3.9.0 and 3.10.1 )

DSPAM is invoked from C with these arguments: dspam --client --user myuser --deliver=innocent,spam --stdout while the message is being fed in through stdin.

I have a copy of the message and the relevant part is here:

--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
.

Hello World, how are you

quoted printable only allows 76 characters per line and as it happens, that line ends with a dot on the 77th character thus putting it on a new line ( what are the odds, right ? )

situation can be reproduced like this: cat messagefile | dspam --client --user myuser --deliver=innocent,spam --stdout

For that message, the output will stop at 'next week to you='.

If I modify the message to stuff the dot manually (..) it will output:

--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
..

Hello World, how are you

Obviously something is wrong. I see that dspam has dot stuffing mechanisms in client.c but it's obviously not working. Is it because of the quoted printable ?

[Stevan Bajic]

I am not really sure if this is a bug or not. Actually the application sending data to DSPAM is responsible for doing the dot stuffing. In your case you use cat to pipe the message to DSPAM. So 'cat' should do the dot stuffing. Correct me if you see that differently.

[polaris1]

Hmm, I saw this in client.c. I thought DSPAM client would take care of that:

* DESCRIPTION
* connect to a dspam daemon socket and attempt to process a message
* this function is called by the dspam agent when --client is specified

/*
* fill buf with partial msg, replacing \n with \r\n
* and do dot stuffing, if needed.
*/

/* take care of dot stuffing \n */
if (message->data[i + 1] && message->data[i + 1] == '.') {
buf[buflen] = '\n';
buflen++;
buf[buflen] = '.';
buflen++;
buf[buflen] = '.';
buflen++;
i += 2;
continue;
}

I haven't really debugged the code, but it seems there is code there that is supposed to do dot stuffing...

[Stevan Bajic]

Yes. DSPAM binary is responsible doing the dot stuffing WHEN sending to a delivery agent OR sending to the DSPAM daemon.

But in your case you feed the DSPAM binary with wrong data. So DSPAM stops processing the message after you just send ONE dot.

It is the same as if you would open a connection to port 25 on your system and post something like this:

< 220 mail.example.com ESMTP Postfix (2.9.3)\r
> EHLO mail.example.com\r
< 250-mail.example.com\r
< 250-PIPELINING\r
< 250-SIZE 52428800\r
< 250-ETRN\r
< 250-STARTTLS\r
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5\r
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5\r
< 250-ENHANCEDSTATUSCODES\r
< 250-8BITMIME\r
< 250 DSN\r
> MAIL FROM:<root@example.com>\r
< 250 2.1.0 Ok\r
> RCPT TO:<stevan@bajic.ch>\r
< 250 2.1.5 Ok\r
> DATA\r
< 354 End data with <CR><LF>.<CR><LF>\r
> Return-Path: <root@example.com>\r
> Delivered-To: stevan@bajic.ch\r
> To: <stevan@bajic.ch>\r
> From: <root@example.com>\r
> MIME-Version: 1.0\r
> Content-Type: multipart/related;\r
> boundary="_004_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_";\r
> type="multipart/alternative"\r
> Subject: Testing\r
> \r
> --_004_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_\r
> boundary="_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_"\r
> \r
> --_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_\r
> Content-Type: text/plain; charset="us-ascii"\r
> Content-Transfer-Encoding: quoted-printable\r
> \r
> Dear Daniela,\r
> \r
> According to Michela, I will send the loading schedule for next week to you=\r
> .\r
> \r
< 250 2.0.0 Ok: queued as 7FDC0A3C170\r
> Hello World, how are you\r
< 502 5.5.2 Error: command not recognized\r
> RSET\r
< 250 2.0.0 Ok\r
> QUIT\r
< 221 2.0.0 Bye\r

Assuming you run Postfix, would you say that this is a Postfix bug? I personally would say that this is NOT a bug.

Now on the other hand if you have a MTA that is getting the above mail and then sends that message to DSPAM over SMTP then the MTA is responsible to do the dot stuffing, while DSPAM is responsible for doing the dot destuffing. When the message is inside DSPAM (the client) and the client is sending the data to the DSPAM daemon (because of the use of --client) then the DSPAM client is doing dot stuffing while the DSPAM daemon is doing the dot destuffing.

[polaris1]

There is no MTA involved here. dspam in --client mode, according to my interpretation of client.c should perform dot stuffing.

If I read that incorrectly, can you tell me what all that code I pasted in client.c is supposed to do ?

[polaris1]

Furthermore, I added this code in client.c in order to make sure the dot stuffing code is working:

r = send(TTX.sockfd, buf+t, buflen - t, 0);
strncpy(sbuf, buf+t, r);
sbuf[r]='\0';
printf ("SENT: %s\n\n\n", sbuf);

And it prints properly the following:

SENT: "_005_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_";
type="multipart/alternative"
MIME-Version: 1.0

--_005_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: multipart/alternative;
boundary="_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_"

--_000_9D6F18858B1E854EBC490A2C24C011D629FA52B099SERVER02Royal_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
..

Hello World, how are you

So obviously the client function detects the lone dot, stuffs it and sends it over properly. There has to be a bug in the daemon code ?

[polaris1]

Yeah, it's definitely within the daemon code where there is no dot stuffing code.

Using TCPDUMP here's the output of the outgoing packets towards the daemon:

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
..

Hello World, how are you
.

and Incoming from Daemon to Client:

Dear Daniela,

According to Michela, I will send the loading schedule for next week to you=
.

Hello World, how are you
.

It fails to stuff it properly on the way back...

[polaris1]

So yeah, I think we can safely call this a bug now.

1. dspam client properly does dot stuffing
2. dspam daemon properly does dot de-stuffing when analyzing the message
3. dspam daemon does not re-stuff the dot when sending the reply back to the client

in client.c we have this line:
while(line != NULL && strcmp(line, ".")) {

which automatically cuts the message off at the first lone dot, unprocessed by #3 above

[polaris1]

When running with --stdout, it's obvious that there should not be any dot stuffing happening, so maybe this behavior is ok after all.

The question is, why is DSPAM concerned about dots as a message end marker, when the daemon replies.

In daemon.c we have:

/* Send a terminating '.' if --stdout in 'dspam' mode */

if (ATX->sockfd_output) {
if (send_socket(TTX, ".")<=0)
goto CLOSE;

and in client.c there is

while(line != NULL && strcmp(line, ".")) {

Why can't that code be removed entirely from daemon.c and the line in client.c turned into:

while(line != NULL) {

It would take care of the problem it seems. What am I missing ?

[Paul van Tilburg]

I wonder what --client really should have to do with dot stuffing? I mean, --client is just about how it is handle in the backend (either run a sole instance or use the client/daemon model).

In my case I use a pipe to forward the (not dot-stuffed message) in combination with --client for scalability reasons, but then suddenly, dot stuffing is not applied and the message gets truncated. Without --client it all works fine. I find this weird and, I hope we can agree on this, at least undocumented behaviour.

[Edwin Eefting]

We also ran against this problem. I definitely think this is a bug:

According to the smtp rfc, the dot stuffing/destuffing is part of the smtp-conversation (http://tools.ietf.org/html/rfc5321#section-4.5.2)

Dspam usually happens AFTER the conversation and should expect whole messages without worrying about any smtp protocol handling. (such as stuffing/destuffing)

There is no need for dspam to know about the special meaning of a single dot on an empty line. (in smtp this in the end-of-message indicator)

When calling dspam without --client everything is fine and somehow magically works, but when using the --client option the bug is triggered somehow. This inconsistent behaviour only contributes to the fact that something is not right.

Tested this with dspam version 3.10.2

Edwin Eefting

Edwin

Edwin Eefting

[Edwin Eefting]

After hours of digging though the code i found the bug and came up with a workaround:

The problem is basically that the deamon removes the dotstuffing, but fogets to add it in the message that is send back. This cause the client to see a single dot and stop reading the message, resulting in a truncated message.

However, if the server would re-add the dotstuffing, the client wouldt remove it, resulting in extra dots send to stdout.

I couldn't find an easy way of fixing in the right way(tm), since i dont know enough of the dspam architecture, and i since we want to use this in production right away the risk of breaking stuff was too big. (it might even involve a rewrite of some code)

So i came with this workaround:

https://github.com/psy0rz/stuff/blob/master/patches/dspam-dotstuffing-workaround.patch

This fixes the client, but fixing the deamon the right way involves some more steps. I think this should be easy to do for the core developers since the source of the problem is known now.

(i've added some FIXME's and comments in the appropriate places)

Edwin Eefting

[James Cloos]

I just noticed this, too.

When the daemon sends the body back to the client, it does not stuff the dots, and the client stops receiving the body once it sees a lone dot.

To fix, the stuffing needs to happen on the return, not just on the send.

[James Cloos]

Argh. I failed to read every reply. Edwin’s patch looks good, but to get it to work I also had to change client.c to triple rather than just double lone dots when sending to the daemon.

[Sergey Kazurov]

Faced a similar problem. We use dspam (3.10.2_3 freebsd version) together with exim. Checking messages is as follows

command = "/usr/local/bin/dspam --innocent --user $local_part@domain -- %u"

Some clients send us messages with attachments from the Outlook 2010 and when the message comes out of the dspam it turns out to be cut off because of the dot symbol at line 3794. Those, the byte stuffing is not applied correctly. As a result, the recipient does not help to open the letter correctly. The source file is called md50002741235.eml, the file after dspam msg1630336.eml. As a result, the recipient does not help to open the letter correctly.
It is possible in any way to fix the problem?

[Eric Broch]

--- ./src/dspam.c 2012-04-11 12:48:33.000000000 -0600
+++ ./src/dspam.new.c 2021-04-12 13:21:58.919951576 -0600
@@ -103,6 +103,40 @@
#define USE_SMTP (_ds_read_attribute(agent_config, "DeliveryProto") && !strcmp(_ds_read_attribute(agent_config, "DeliveryProto"), "SMTP"))
#define LOOKUP(A, B) ((_ds_pref_val(A, "localStore")[0]) ? _ds_pref_val(A, "localStore") : B)

+buffer * stuff_dot(char * message);
+/
+ buffer * stuff_dot(char * message);
+
+ DESCRIPTION
+ Stuff '.' where necessary for SMTP Protocol
+
+ INPUT ARGUMENTS
+ char * message
+
+ RETURN VALUES
+ Returns buffer * msg
+
+ Author
+ Eric C. Broch
+
+/
+buffer * stuff_dot(char * message) {
+
+ buffer * msg;
+ int i = 0;
+
+ msg = buffer_create(NULL);
+ while(i<strlen(message)) {
+ if (buffer_ncat(msg,&message[i],1)) return NULL;
+ if (i == strlen(message)-1) break;
+ if ((message[i] == '\n') && (message[i + 1] && message[i + 1] == '.')) {
+ if (buffer_cat(msg,".")) return NULL;
+ if (buffer_ncat(msg,&message[++i],1)) return NULL;
+ }
+ i++;
+ }
+ return msg;
+}

int
main (int argc, char *argv[])
@@ -977,7 +1011,18 @@
(result == DSR_ISSPAM) ? "SPAM" : "INNOCENT");

if (mailer_args == NULL) {
- fputs (message, stream);
+ / calling program: dspamc /
+ if (stream == ATX->sockfd) {
+ / stuff dots before sending back to dspam client /
+ / otherwise trunction of email can occur, Eric Broch 04-12-2021 /
+ buffer * msg = stuff_dot(message);
+ fputs(msg->data,stream);
+ buffer_destroy(msg);
+ }
+ / calling program: dspam /
+ else {
+ fputs(message,stream);
+ }
return 0;
}

--- ./src/client.c 2012-04-11 12:48:33.000000000 -0600
+++ ./src/client.new.c 2021-04-12 13:14:45.152130732 -0600
@@ -228,6 +228,14 @@
exitcode = 99;
}
} else {
+ / de-stuff dots from server, 04-12-2021, Eric C. Broch, 7 lines /
+ if((line[0] && line[0]=='.') && (line[1] && line[1]=='.')) {
+ size_t i, len = strlen(line);
+ for(i=0;i<len;i++){
+ line[i]=line[i+1];
+ }
+ line[len-1]=0;
+ }
printf("%s\n", line);
}
free(line);