e-mail code reorganization

[Robert Devantier]

I've been looking at the code while trying to figure out why it won't send e-mails to me. I have an Exchange server and tried the fix in a previous topic but that isn't working.

At the end of the script, wouldn't it make sense to issue the firewall commands inside the if condition to send e-mail? Otherwise you're opening the firewall port for SMTPout on the ESXi box every time, which may not be what the system owner desires.

Also, any word on the 3.1.0 version that was going to be released at the end of May? I just stumbled onto this script and it works more than twice as fast at backing up my very large VMs as compared to vBackup. A single VM takes 2 hours using vBackup, and only took 30 minutes with this script!

Anyway, change the script to something like this:

# We send the e-mail

if [[ $SEND_EMAIL == 1 ]]
then

        OFRESPONSE=$(openFirewall "$smtpport")

        sleep 2
        esxcli network firewall ruleset set --ruleset-id=SMTPout --enabled=true
        esxcli network firewall refresh
        ...

[Robert Devantier]

Also, I found two more formatting issues:

1. About 5 instances where the "br/" should be changed to "br /" for the XHTML compatability.

2. I noticed a lot of "\n" things appearing in the HTML file string that weren't being translated into newlines. This is under ESXi 5.5 1746018. If I changed all the "echo ..." to "echo -e ..." then they translated correctly into the file and the resulting HTML looked very good in the browser. The other option is to do echo $'Hello\nWorld' instead.
   -e flag, which "enables interpretation of backslash escapes".
   $' - Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.

[33HOPS]

Thank you Robert, we will take all your comments into account. There is for sure some minor bugs like the ones you are pointing at.

[Robert Devantier]

Just another set of "fixes" that would help, working from the 4.0.2 source...

1) The --smtp-auth switch is not documented in the README.txt or the code itself. I have to use this against my Exchange server, it doesn't support the AUTH LOGIN PLAIN command.

2) A few corrections to the mail code. I capitalized the HELO, ELHO, and QUIT commands just because it looks better. I also took the spaces out after the colons, since RFC821 didn't officially show them, and re-ordered some of the DATA commands. I took the Date line from Dave Brave's post and added it. More importantly, I added the if/then logic around the AUTH part so that when smtp-auth is set to none it doesn't submit the group of auth commands. Oh, and I made the one line a \r\n.\r\n, even though it worked as-is, and then added the \r\n to the end of QUIT since RFC821 showed that as well.

        mail_input() {
        echo -ne "HELO ${HOSTNAME}\r\n"
        echo -ne "EHLO ${HOSTNAME}\r\n"

        if [[ "${smtpauth}" != "none" ]]
        then
            echo -ne "AUTH LOGIN\r\n"
            echo -ne "$( base64_encode $smtpusr )\r\n"
            echo -ne "$( base64_encode $smtppwd )\r\n"
        fi

        echo -ne "MAIL FROM:<${mailfrom}>\r\n"
        echo -ne "RCPT TO:<${mailto}>\r\n"
        echo -ne "DATA\r\n"
        echo -ne "Date: `date '+%a, %d %b %Y %H:%M:%S %z'`\r\n"
        echo -ne "From: <${mailfrom}>\r\n"
        echo -ne "Subject: ${subject}\r\n"
        echo -ne "To: <${mailto}>\r\n"
        echo -ne "Content-Type: text/html\r\n"
        echo -ne "\r\n"
        echo -ne ${emailHTMLStr//\\n/$newline}"\r\n"
        echo -ne "\r\n.\r\n"
        echo -ne "QUIT\r\n"
        }

I still cannot get it to send emails from my hMailServer, which works with no problem when I use Blat. I have to dig into the configuration and make sure I've got all the rules correct, but it does work with my Exchange server.

Using Exchange:

Opening port 25 for SMTPout service...
220 mail.xxx.yyy Microsoft ESMTP MAIL Service ready at Mon, 30 Jun 2014 09:15:31 -0400
250 mail.xxx.yyy Hello [a.b.c.d]
250-mail.xxx.yyy Hello [a.b.c.d]
250-SIZE 25600000
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH
250-8BITMIME
250-BINARYMIME
250 CHUNKING
250 2.1.0 Sender OK
250 2.1.5 Recipient OK
354 Start mail input; end with <CRLF>.<CRLF>
250 2.6.0 <f360cee1-0225-46bd-93fb-fde3646a8434@mail.xxx.yyy> [InternalId=108183827] Queued mail for delivery
221 2.0.0 Service closing transmission channel
Firewall rule SMTPout closed.

Using hMailServer:

Opening port 25 for SMTPout service...
220 mail.zzz.xxx.yyy ESMTP
250 Hello.
250-mail.zzz.xxx.yyy
250-SIZE 20480000
250 AUTH LOGIN PLAIN
250 OK
250 OK
354 OK, send.
Firewall rule SMTPout closed.

[33HOPS]

You are free to make changes to the code, that's the magic of Open Source.
In regards to your hMailServer I would try to write the protocol line by line by telnetting at port 25 and see how this server likes the SMTP conversation to be done.

[Robert Devantier]

This is a great script, I really like it! Just trying to help you make it better, hope that's OK.

I did try a telnet, and the problem is that it works to my hMailServer and I get the e-mail. For some reason it is not liking the netcat method that you are using in the script. I get the same number of responses back from the mail server (the 250s, 354, etc) so it seems like it is getting the right stuff. I even went as far as changing the one echo line with the body of the message to just be: echo -ne "This is a test.\r\n" and that didn't work, yet it worked from telnet.

This is the output, with personal/machine information changed for privacy reasons.

220 mail.xxx.yyy ESMTP
HELO vSphere1
502 Use HELO/EHLO first.
EHLO vSphere1
250-mail.xxx.yyy
250-SIZE 20480000
250 AUTH LOGIN PLAIN
MAIL FROM:<myemail@xxx.yyy>
250 OK
RCPT TO:<myemail@xxx.yyy>
250 OK
DATA
354 OK, send.
From: <myemail@xxx.yyy>
Subject: Telnet test
To: <myemail@xxx.yyy>
Content-Type: text/html

This is a test.
.
250 Queued (47.172 seconds)
QUIT

[Robert Devantier]

Found the fix! Netcat was being over-run with too many commands in a row. I added two sleep commands and it now sends the e-mail via both my Exchange server and my hMailServer. I also verified it works on one of my larger vSpheres with over 30 VMs with a --backup-type=running command (took 7 minutes to generate via test-mode and 133 seconds to queue, meaning large emailHTMLStr content) so I think the sleep values should be good for most everybody.

        mail_input() {
        echo -ne "HELO ${HOSTNAME}\r\n"
        echo -ne "EHLO ${HOSTNAME}\r\n"

        if [[ "${smtpauth}" != "none" ]]
        then
            echo -ne "AUTH LOGIN\r\n"
            echo -ne "$( base64_encode $smtpusr )\r\n"
            echo -ne "$( base64_encode $smtppwd )\r\n"
        fi

        echo -ne "MAIL FROM:<${mailfrom}>\r\n"
        echo -ne "RCPT TO:<${mailto}>\r\n"
        echo -ne "DATA\r\n"
        echo -ne "Date: `date '+%a, %d %b %Y %H:%M:%S %z'`\r\n"
        echo -ne "From: <${mailfrom}>\r\n"
        echo -ne "Subject: ${subject}\r\n"
        echo -ne "To: <${mailto}>\r\n"
        echo -ne "Content-Type: text/html\r\n"
        echo -ne "\r\n"
        echo -ne ${emailHTMLStr//\\n/$newline}"\r\n"
        sleep 3
        echo -ne "\r\n.\r\n"
        sleep 3
        echo -ne "QUIT\r\n"
        }

[33HOPS]

Netcat does not care about the size of the data you are passing to it. If you had to add these sleep commands for your hMailServer to read the protocol input then that means there is a timiout issue at the time of reading the protocol lines. Almost every SMTP server lets you tweak this timeout between lines so you should tweak your server. Adding sleep comands is just a hack and may very well work the opposite way with some other SMTP servers.

[Robert Devantier]

Netcat might not care, but there is an over-run happening somewhere between netcat and the SMTP server. Since others have had problems getting e-mails to work, this might be good information for them as well. I found no settings in hMailServer that affect the timeout between lines, and the hardware my SMTP server is on is very fast and highly underutilized.

I don't think a 2 or 3 second sleep command is going to cause any SMTP server to close the connection before the completion of the stream. In looking for solutions to my problem (different methods to send email via bash without using sendmail) I saw many script options that had sleep delays around certain commands. So other script writers have noticed this issue as well.

I'm just happy I have it working now. With the addition of the 10GbE card I just installed in my FreeNAS box, my backups are now flying. What once took 18.5 hours using vBackup were down to 5 hours with your script, and now will be even less with the upgraded network bandwidth!

[Zedde]

I keep getting this

Apr 28 21:15:52 - exim[12112]: 2015-04-28 21:15:52 SMTP protocol violation: synchronization error (input sent without waiting for greeting): rejected connection from H=ma.inet [192.168.] input="EHLO master\r\n"
Apr 28 21:13:43 - exim[11842]: 2015-04-28 21:13:43 SMTP protocol violation: synchronization error (input sent without waiting for greeting): rejected connection from H=ma.inet [192.168.] input="HELO master\r\nEHLO master\r\n"

I have added sleep after HELLO and EHLO, tried to # out HELO as you can see in the first line.

[33HOPS]

It looks like a timing issue. You need PIPELINING activated in your SMTP server. Read this: http://33hops.com/blog_xsibackup-smtp-client.html

[33HOPS]

We will add an argument --smtp-delay=seconds to allow this type of workaround, it should not be necessary in an e-mail server that implements PIPELINING though.