#1456 Curl SMTP client CRLF and dot stuffing doesn't work

closed-fixed
smtp (2)
5
2014-12-12
2014-11-25
bra
No

Hi,

I'm trying to send a simple e-mail with curl. SMTP has two caveats here: dot stuffing and CRLF, I've tried to test these in my sample e-mail.
This is the code, which is mostly the smtp example from curl source:
http://pastebin.com/0JtwGTqx

For the options, I've just added curl_easy_setopt(curl, CURLOPT_CRLF, 1L); to do CRLF conversions, and replaced \r\n in payload_text to see whether it works or not.

Bug #1: this example sends LF line endings on the wire, so my SMTP server refuses the e-mail. Shouldn't CURLOPT_CRLF convert them to CRLF?
Bug #2: dot stuffing doesn't happen. The lines with a dot in the first character aren't get prepended with an extra dot, so the transaction here fails (because of the lone LF.LF). If think an SMTP client should do this and not let the client take care of it.

Could you please investigate these?

Thanks,

$ curl -V
curl 7.39.0 (amd64-portbld-freebsd10.0) libcurl/7.39.0 OpenSSL/1.0.1i zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS Largefile GSS-API SPNEGO NTLM NTLM_WB SSL libz TLS-SRP

Discussion

  • Steve Holme

    Steve Holme - 2014-11-25

    Hi,

    Thank you for your report.

    I think you have found a bug in the documentation!

    Looking at the code in transfer.c that performs the LF to CRLF conversion it is contained within the else block of:

    if(conn->handler->protocol & PROTO_FAMILY_SMTP) {
      result = Curl_smtp_escape_eob(conn, nread);
      if(result)
        return result;
    }
    else
    

    The dot stuffing happens in Curl_smtp_escape_eob() which checks for CRLF. replacing it with CRLF..

    I will need some clarification from other members of the curl development team, as to which protocols are supposed to support the conversion but I don't believe SMTP does given the above and my experience with curl in this area.

    If you remove your usage of CURLOPT_CRLF and transfer a file that contains CRLF you should find that dot stuffing works.

    Kind Regards

    Steve

     
    • bra

      bra - 2014-11-26

      Hi,

      First of all, thanks for taking care of this.
      I think CRLF conversion is a must for SMTP. For example, all of (at least what I know) LMTP/IMAP/POP servers which use the maildir format store the messages in unix (LF) format. Sending these out with SMTP requires conversion in my application, which (converting to and from wire or network format) should be the task of the network library -here the SMTP client, curl.
      If the infrastructure is already there, I can't see a good reason not to do this. :)

      Thank you very much.

       
  • Steve Holme

    Steve Holme - 2014-11-25

    In the meantime I have updated the documentation to reflect what the code does.

     
  • Steve Holme

    Steve Holme - 2014-11-26

    Yet another update...

    This was annoying me a little so I have spent the last 40 minutes working on a fix!

    I will perform some final testing tomorrow, review and push if all is good.

    Kind Regards

    Steve

     
    Last edit: Steve Holme 2014-11-26
  • Steve Holme

    Steve Holme - 2014-11-26
    • assigned_to: Steve Holme
     
  • Steve Holme

    Steve Holme - 2014-11-26

    I have a pushed my implementation of this as commit 4bd860a001

    It is a little different from last nights quick fix as I found a memory leak in my code and fixed a few other minor issues in the next commit.

    I have also updated the documentation accordingly.

    I have a new test case for this as well but will push that in the morning after some shut eye ;-)

    If you build libcurl yourself you can apply the git format patch by following the link above (and appending .patch to the url to download the associated .patch file). Alternatively the next nightly archive will contain this.

     
    Last edit: Steve Holme 2014-11-26
  • Steve Holme

    Steve Holme - 2014-11-26

    I will close the issue if you are able to confirm it works for you / once I have pushed my test case and verified the result.

     
  • Steve Holme

    Steve Holme - 2014-11-26
    • status: open --> open-confirmed
     
  • Steve Holme

    Steve Holme - 2014-11-27

    I pushed the additional test case (test941) this morning as commit c5399c827d which looking at the next ran autobuild verifies the fix.

     
  • Steve Holme

    Steve Holme - 2014-11-27

    Thanks again for your report.

     
  • Steve Holme

    Steve Holme - 2014-11-27
    • status: open-confirmed --> closed-fixed
     
    • bra

      bra - 2014-11-28

      Hi,

      Thanks for the quick fix for the CRLF issue, that works now.
      BTW, the example program still can't send the e-mail, because the dot stuffing doesn't work.

      Here's the wireshark output from the session:
      DATA
      354 End data with <CR><LF>.<CR><LF>
      Date: Mon, 29 Nov 2010 21:54:29 +1100
      To: recipient@example.org
      From: sender@example.org(Example User)
      Message-ID: dcd7cb36-11db-487a-9f3a-e652a9458efd@rfcpedant.example.org
      Subject: SMTP example message

      .The body of the message starts here.

      .
      It could be a lot of lines, could be MIME encoded, whatever.
      Check RFC5322.
      .
      250 2.0.0 Ok: queued as B82944109EF
      502 5.5.2 Error: command not recognized
      502 5.5.2 Error: command not recognized
      502 5.5.2 Error: command not recognized
      QUIT

      Could you please fix that too?

      Thanks,

       
      • Steve Holme

        Steve Holme - 2014-11-28

        I'm a little confused as the dot stuffing has worked for a long time (See test 901) and I haven't changed that code (apart from the different internal buffer it now looks at).

        The reason it wasn't working for you previously is that the dot stuffing looks for CRLF. and not LF. as was happening before I added support for --crlf to perform the conversion. The dot stuffing now happens after that conversion so intenally your mail will be CRLF. and the dot stuffing should work ;-)

        I don't run the test suite personally as I run under Windows but my own test files are working okay ;-) I have dot stuffing with CRLF and dot stuffing with LF as well as plain emails without any dot stuffing.

        However, I will investigate this further tonight to see if there are any other issues in the dot stuffing code that could prevent the example working for you.

        As such I will re-open this issue.

         
        Last edit: Steve Holme 2014-11-28
        • Steve Holme

          Steve Holme - 2014-11-29

          Just a quick update...

          I don't build against libcurl for use in an application at home but instead build curl so my testing tonight has been a little different from yours.

          I used the following command line:

          curl -v -ssl -k smtp://smtp.example.com -T C:\curl\mail-lf.txt --crlf --mail-rcpt recipient@example.org --mail-from sender@example.org

          Whilst debugging transfer.c and smtp.c I can see my strings are as follows (I have removed the headers and body separator for clarity):

          As read from my file:

          ".The body of the message starts here.\n.\nIt could be a lot of lines, could be MIME encoded, whatever.\nCheck RFC5322.\n.\n"

          After LF to CRLF conversion:

          ".The body of the message starts here.\r\n.\r\nIt could be a lot of lines, could be MIME encoded, whatever.\r\nCheck RFC5322.\r\n.\r\n"

          After calling Curl_smtp_escape_eob() to do the dot stuffing:

          "..The body of the message starts here.\r\n..\r\nIt could be a lot of lines, could be MIME encoded, whatever.\r\nCheck RFC5322.\r\n..\r\n"

          Do you have the same issues if you use curl?

          Kind Regards

          Steve

           
          Last edit: Steve Holme 2014-11-29
  • Steve Holme

    Steve Holme - 2014-11-28
    • status: closed-fixed --> open-confirmed
     
  • Steve Holme

    Steve Holme - 2014-12-12

    After some discussion on the libcurl mailing list, we think we have found the issue, which would only affect libcurl based applications that a) used a read function and b) where the EOL character(s) were at the end of each buffer.

    As such I have pushed commit f0ecdd04d3

     
  • Steve Holme

    Steve Holme - 2014-12-12
    • status: open-confirmed --> closed-fixed
     
  • Steve Holme

    Steve Holme - 2014-12-12

    The fix doesn't appear to broken the existing dot stuffing tests - although it would be good if we could add a libcurl test to also verify this.

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks