E-MailRelay / Support Requests / #75 Emailrelay as a Self hosting Email server on OpenWrt router

Graeme Walker - 2022-10-01

I use emailrelay for a self-hosted mailserver, so I would never say just use postfix!

In my experience you will need to use an address-verifier script to cope with botnet spam attacks. With a simple whitelist the bots will just keep trying different names and never give up. For example, a script can completely block source IP addresses at the firewall if the same address tries too many bogus recipients, or it can introduce a tar-pit delay for IP addresses that it has never seen before. In fact, always using a short delay might be a good idea if email throughput is not critical.

The pop-by-name feature does require something external (like a filter script) to do the copying into the separate users' sub-directories. I have included the emailrelay-filter-copy utility in the distribution that can be used out-of-the-box, but it copies all emails to all users, which is not what you are asking for. Writing a filter script to do pop-by-name copying based on the envelope "To:" address should not be too difficult, but note that any bugs or corner-cases would be privacy-busting.

You shouldn't be too worried about having multiple instances of emailrelay running. The code pages will be shared and there is not much held in data pages -- the emails themselves, for example, are never held in memory. Note that for very resource-limited targets you should compile with -Os and consider using -DG_SMALL (ie. ./compile.sh CXXFLAGS="-Os -DG_SMALL").

tbc...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2022-10-01

\1. address whitelist

Probably not a good idea (as above), but it would be easy enough to implement with a command-line syntax something like "--address-verifier=accept:/etc/whitelist".
I might consider writing that as a patch, but probably not a main-line feature.

\2. pop-by-name

Write a script to examine the envelope "To:" address(es) and map to a pop-by-name directory. Copy the content file first, then copy the envelope file with a ".tmp" extension, and finally rename the copied envelope file (atomically) to remove the extension.

Copying the content file is optional because the emailrelay POP server will look in the parent directory if the content file is not in the pop-by-name directory. However, there is then a risk that the envelope becomes orphaned if the content file is lost from the parent directory. So on unix-like systems it is better to hard-link the content file into the sub-directory.

If the message is being deleted from the main spool directory then the filter script should use an exit code of 100.

Ideally the script should be callable as a filter script or as a cron job so that it can be run from cron from time-to-time as a backup.

\3. export to maildir

That's a good idea. You can use a filter script to export into a 'maildir' system very easily. You just need to copy or move the content file into the 'tmp' subdirectory and atomically rename it into 'cur'. In practice there is no need to change the filename (YMMV).

\4. mx routing

That has always been out of scope for emailrelay. To fudge it you would use a client-filter to do the MX lookup, move the message file into a single-use spool directory and run "emailrelay --as-client" directly from the client-filter.

\5. multiple instances

See above

\6. pop/smtp auth file

Yes, just use the same file for both. The 'server' prefix is required.

\7. 1 reply-to

You need to set a "Reply-To:" content header. If your email client can't be configured to set this then there is a "set-from" example script in the distribution that edits the "reply-to" address. However, it doesn't add the header if it is missing so it would need some tweaking.

\7.2 multiple client auth entries

No, because that assumes some sort of routing (ie. question 4).

\8. local routing

Yes, you can use the 'local mailbox' feature of the address verifier. Any 'local' recipients are listed separately in the envelope file so that they are not used when the email is forwarded and a copy of the email is created with a ".local" extension on both the content file and envelope file.

You just need to check for a ".local" file in a filter script (and/or as a cron job) and move and rename the files as required using the same content-first procedure as above. Parse the envelope file looking for "X-MailRelay-To-Local:" lines.

(edited to fix markup)

Last edit: Graeme Walker 2022-10-03

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Sergey Ponomarev - 2022-10-01
  
  Thank you for the detailed reply,
  
  You see, my main goal is to make it with zero configuration if
  possible. It's great that we can use scripts and change behaviour but
  even more important to have sane defaults.
  Ideally just install the package and start reading your mail.
  
  The "--address-verifier=accept:/etc/whitelist" would be great. As an
  alternative this file can be just the emailrelay.auth.
  
  If the pop/smtp auth file can be the same then this may simplify
  configuration. In order to make configuration even easier the ER may
  just always load /etc/emailrelay.auth by default.
  Then we can avoid the need to specify two options --server-auth and
  --pop-auth. Even if some account is a send only that's not a big deal
  if it has the pop access too.
  
  Speaking about the /etc/emailrelay.auth format: I don't see a reason
  to have the server prefix. If it can be omitted that may simplify
  things.
  Also it would be better to have the .txt or .ini prefix like
  emailrelay.txt so that windows users can just edit it with Notepad
  without problems.
  But for the smtp --client-auth file I think it would be better to have
  a separate file. It also can be put to some location and loaded by
  default.
  
  You see, editing some existing file is easier than creating a new one.
  
  Another problem is how to make a GUI for this. For example the openwrt
  package has an init.d script that just converts UCI lines to the
  program lines e.g.
  
  config emailrelay 'server'
  option mode 'server'
  option port '25'
  option remote_clients '1'
  
  Converts to
  emailrelay --as-server --port 25 --remote-clients
  
  But how to create users from UCI? I see the only option is to generate
  the emailrelay.auth file to /tmp and use it. But the /tmp is not a
  safe place to put creds.
  Or maybe implement a new option like --server-auth-lines "user1 pass,
  user2 pass". This is still not safe because the options are seen from
  ps but much easier to generate.
  
  The mx routing is definitely not that easy to implement as a filter.
  Or even if implemented then it may not be that speed effective
  especially for DNS cache.
  The local routing also seems like not that easy to make.
  I checked sources and found bin/emailrelay-submit.sh.in:
  
  An example script that reads an e-mail message from stdin and
  
  deposits it into
  a sub-directory of the E-MailRelay spool directory depending on the "To:"
  address. This could be used with an E-MailRelay POP server using the
  "--pop-by-name" option so that messages get routed appropriately.
  
  This is exactly what I want to have out of the box. But internally
  this script seems not effective. I don't believe it will work on
  OpenWrt with BusyBox ash and it even uses perl in one place (possibly
  can be replaced with awk).
  
  Maybe something built-in can do the work faster.
  
  Today I found the https://github.com/viridIT/vSMTP which is an SMTPd
  written in Rust and it has conception of "transports"
  https://github.com/viridIT/vSMTP/tree/develop/src/vsmtp/vsmtp-delivery/src/transport
  The "forward" is for "smarthost" but the delivery transport actually
  makes the MX lookup.
  
  Interesting is that it also has some DKIM signature checker built in.
  I believe this also should improve speed.
  
  With a simple whitelist the bots will just keep trying different names and never give up.
  
  This is probably only a problem if some bot is targeting specifically
  my domain. Maybe this is a problem for big sites but not sure if I
  need to someone.
  
  For example, a script can completely block source IP addresses at the firewall if the same address tries too many bogus recipients, or it can introduce a tar-pit delay for IP addresses that it has never seen before. In fact, always using a short delay might be a good idea if email throughput is not critical.
  
  Yes, some built-in smart IP firewall would be great because only a
  small portion of users will bother to configure fail2ban.
  
  Thank you
  
  (edited to remove email chain)
  
  Last edit: Graeme Walker 2022-10-03
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2022-10-03

I like the idea of a streamlined OpenWRT installation, but I'm still not clear about the end goal.

Extrapolating from what you have said so far it is to have a public-facing server with a recipient whitelist and DNSBL; route incoming messages to pop-by-name sub-directories; have another emailrelay instance (?) for outward traffic to a single smarthost account, with authentication of local users and a filter to do Reply-To shenanigans; or perhaps even connecting directly to downstream SMTP servers using MX lookups; and route messages directly if outgoing messages are sent to another local user. And do all that with a minimalistic command-line and no run-time scripts or helpers?

Not all of that will be possible while retaining compatibility with the current program, so it would need to be built as a separate executable.

Anyway, to address some of your other points...

I don't rule out implementing a recipient whitelist if it really helps, but I was thinking it would be a simple list of recipient addresses. For an authentication secrets file to double-up as a whitelist it would have to pick out the third field of every 'server' line and add the domain-name.

The --server-auth and --pop-auth options can point to the same file (as discussed elsewhere) but note that the --server-auth option is slightly more than a pointer to the secrets file, it also enables server authentication (ie. the SMTP AUTH extension). The reason for the "server" prefix is to allow entries of different types in the same file; currently only "client" and "server" but at one time also "pop". I'm not sure how removing it would help: you would then force the client and server secrets to be in separate files, it would break existing users, and you would still need a 'type' field to define the encoding.

I don't understand why having default file locations and the files loaded by default is important. As things stand you can have empty or commented-out files created at install-time (as is done for /etc/emailrelay.conf and emailrelay.auth on windows) and you chose to use them with a command-line option that gives the full path. If the path is baked in then you lose the flexibility to, for example, choose the filename extension as ".auth" or ".txt"! But anyway, I would be okay with doing this as a build-time option so that some command-line arguments could be baked in via the configure script.

Populating the authentication secrets files from UCI is not something I can advise on, but perhaps on the server side at least the problem could be side-stepped by using PAM: emailrelay could read the password database at startup to get a list of users, create pop-by-name directories for each, generate a recipient whitelist by appending the "--domain" value, and then use PAM to do the POP and SMTP server authentication. (I noticed that the OpenWRT makefile explicitly disables PAM authentication. I wonder why that is.)

The emailrelay-submit.sh is an example script so I don't make any claims about it being production quality. The only change in 15 years has been to change from awk to perl (!) because awk is sometimes only installed as "nawk" or "gawk". I think I could be persuaded to add a built-in filter to do something similar to emailrelay-submit.sh or emailrelay-filter-copy.

I have always kept any kind of external routing (MX lookups etc) out of scope for emailrelay because of its potential complexity and because of the privacy implications for getting it wrong. Perhaps I am being overly cautious. As long as routing can be done by just looking at the envelope (what does that Rust code do?) and not having to parse the content headers (eg. for Bcc) then I suppose I should consider it. An easy short-term fix would be to allow envelope files to contain a forwarding address override. An external client-filter could then make a separate copy of the message for each downstream server and deposit the result of the MX lookup into each envelope file. For the new program (emailrelay-openwrt?) the equivalent processing, including the MX lookup, could be done internally. I don't think there is an API for doing MX lookups so this would have to be done with DNS messaging to a configured DNS server (as the DNSBL code already does).

To summarise the possible short-term changes to emailrelay:

allow command-line options to be baked in from the configure script

add a forwarding address override field to the envelope file

write a built-in filter for copying, like emailrelay-submit.sh

write a built-in address verifier to read a recipient whitelist file

What do you think?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Ponomarev - 2022-10-03

Graeme, thank you for the good and detailed reply.

not clear about the end goal.

In ideal world how I see it the end goal looks like just start "emailrelay" without any args (or with one --mailserv flag) and it will listen for incoming mail on 25, listen for outcoming on 587 and give POP on 110. It will automatically find tls certs. The only needed configuration is mailboxes and passwords for them. Traditionally the PAM worked here but anyway not a good idea to use the same password from mail and system's user.
Also we need for some Web UI but this is out of scope of the ER. Some spam and antibot protection is also something that is needed.

Right now to achieve this I need to learn documentation and how email works. And configure some addr verifier and filters. I know it's cool to be flexible but I wish to have some mercy for users.

You see, all the mail servers were created with unix philosophy with many config options to handle various scenarios and for smart sysadmins. Honestly, the ER is probably the only server that tries to by user friendly.

allow command-line options to be baked in from the configure script

I guess that's not so important. The idea that I had is just to minimize cmd args and just load from the one file pre-specified file by default.

I checked and debian package creates the /etc/emailrelay.conf file by default. So if I'll write an instruction I can just say "open the file and change that line".
But it doesn't create the /etc/emailrelay.auth but instead only /etc/emailrelay.auth.template. So now I have to say "copy the template to auth and only then edit it". I.e. one additional manual step.

For the openwrt package things are different. It creates the /etc/emailrelay.auth but the /etc/emailrelay.conf is not created but instead used UCI config /etc/config/emailrelay.
And the /etc/emailrelay.auth is not specified to be used by default. So I have to say in the instruction "edit the auth file and then in the UCI config you need to specify it". I.e. again one additional step.
But this can be easily solved: I'll just send a PR for the /etc/config/emailrelay and the /etc/emailrelay.conf will be pre-specified for both --server-auth and --pop-auth.

add a forwarding address override field to the envelope file

Probably that would be useful for those who forwards emails. I want to be independent and deliver it directly by MX lookup.

write a built-in filter for copying, like emailrelay-submit.sh
write a built-in address verifier to read a recipient whitelist file

Yes, please. If they can built-in then this also makes speed better.
If you need any help on this I can try contribute. But it's not comfortable to do on the SourceForge and I have to create some patches.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Ponomarev - 2022-10-13

Hi again, speaking about delivery here I found a sendmail that can deliver directly to SMTP https://github.com/richfelker/mxclient

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2022-10-17

If you want a single process to do incoming mail on port 25 and outgoing submission on port 587 together with MX routing then that is significantly different from today's emailrelay -- it's effectively two emailrelays in one, with one spool directory for incoming mail and another for outgoing. It will therefore need to be a new program within the emailrelay project, and while I am happy to take that on for this winter I am currently working towards a v2.4 release.

Some of my comments above were for simple changes I could make to the existing code that might help in the short term. In particular, the forwarding address override in the envelope file would allow some experimentation of routing with MX lookups; an external client filter could do the lookup and stuff the result in the address override field, with almost no change to the emailrelay code. I might squeeze that into the 2.4 release in any case.

We should probably continue this discussion via email.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Ponomarev - 2022-10-21

I may try to contribute to the project but not sure how to do that. Just send a patch?
There is already an unofficial clone repo on GitHub https://github.com/aclemons/emailrelay so maybe you can switch to Git to make it easier?

I'm pretty sure that I can found other people who can also be interested in the light self hosted smtp.

The built-in addr verifier is something that I can make for sure. And it will reject spam attacks quickly without starting shell script.

The emailrelay-submit.sh sorting is something that I also can implement but not so critical for self hosting.

But the basic MX delivery would be just great because makes mail finally private . I would

The https://github.com/richfelker/mxclient is interesting because uses DANE which is kind of TLS without CA but just based on DNS. This is very useful for embedded devices where it's hard to store all CA certs. But what is more important is that it can work with unofficial DNS like Namcoin .bit domains, KadNode or even Tor's .onion.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2022-10-22

Okay, I will look into switching the sourceforge repository to git and by all means work from there.

Please take into consideration that emailrelay has a very particular scope, so some of the changes we have discussed should be for a new binary within the emailrelay project. This would give you freedom to be linux-only, for example, or use additional third-party libraries.

FYI I have started work on adding some basic routing capabilities. It might remain an 'experimental' feature within the classic emailrelay program, but it should be useful for prototyping your work.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2022-10-22

I have populated the sourceforge git repository:

git clone https://git.code.sf.net/p/emailrelay/git emailrelay

Is that useful?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-01-03

So a lot of your suggestions are now implemented in the 2.5 development branch (svn V_2_5), as documented in the NEWS file.

Specifically:

basic routing made it into v2.4

the main binary can be hard-linked as emailrelay-submit (busybox style)

multiple configurations in one process allow both 'in' and 'out' operation simultaneously

delivery is a thing, so messages can be delivered to individual mailboxes

built-in address verifiers work without any special configuration

outgoing messages can be turned around for local delivery, not via the smarthost

Let me know what you think.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Ponomarev - 2023-01-03

Thank you, this is a great news! I'll try to test it locally as soon as possible and going to send a PR with the version upgrade to OpenWrt maintainers.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-01-04

Okay but note that the 2.5 branch is not release quality yet. For one thing the documentation is still mostly for 2.4.

FYI, I have been testing the new features with something like this:

./emailrelay --log --verbose --log-time --no-daemon --in-spool-dir @app/in --in-port 10025 --in-domain example.com --in-address-verifier allow: --in-filter deliver: --in-pop --in-pop-port 10110 --in-pop-auth /pam --in-pop-by-name --out-spool-dir @app/out --out-port 10587 --out-domain example.com --out-address-verifier local: --out-local-delivery-dir @app/in

By default the address verifiers look for entries in /etc/passwd with a uid of 512 or more, but they can be giving a uid range like "local:1000-1001". The delivery filter also checks /etc/passwd but it does not care about the uid.

In testing you should see that outgoing messages for alice@example.com (where alice is in /etc/passwd) should end up in the out/alice mailbox and be available via POP using alice's normal login password. Messages for nobody@example.com or alice@nowhere.com will be queued up in the "out" spool directory in the normal way.

Incoming messages for alice@example.com will also end up in the in/alice mailbox, but messages for nobody@example.com or alice@nowhere.com will be rejected by the SMTP dialog. Without the --in-address-verifier incoming messages for nobody@example.com or alice@nowhere.com would be accepted and go into a 'postmaster' mailbox, which could be a symlink to another mailbox.

I hope that's the sort of thing you had in mind.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Hi Greame,
I compiled the v2.5 and runned with the command that you provided and tried to send a mail but got '452 failed":

emailrelay: info: GSmtp::ServerSend: tx>>: "220 stokito-serv -- E-MailRelay V2.5dev2 -- Service ready"
emailrelay: info: GSmtp::ServerProtocol: rx<<: "EHLO [127.0.0.1]"
emailrelay: info: GSmtp::ServerSend: tx>>: "250-stokito-serv says hello"
emailrelay: info: GSmtp::ServerSend: tx>>: "250-VRFY"
emailrelay: info: GSmtp::ServerSend: tx>>: "250-PIPELINING"
emailrelay: info: GSmtp::ServerSend: tx>>: "250 8BITMIME"
emailrelay: info: GSmtp::ServerProtocol: rx<<: "MAIL FROM:<stokito@stokito-serv>"
emailrelay: info: GSmtp::ServerSend: tx>>: "250 sender <stokito@stokito-serv> OK"
emailrelay: info: GStore::NewFile: content file: /home/stokito/spool/out/emailrelay.49933.1673124642.2.content
emailrelay: info: GSmtp::ServerProtocol: rx<<: "RCPT TO:<stokito@disk.stokito.com>"
emailrelay: info: GSmtp::ServerSend: tx>>: "250 recipient <stokito@disk.stokito.com> OK"
emailrelay: info: GSmtp::ServerProtocol: rx<<: "DATA"
emailrelay: info: GSmtp::ServerSend: tx>>: "354 start mail input -- end with <CRLF>.<CRLF>"
emailrelay: info: GSmtp::ServerProtocol: rx<<: [message content not logged]
emailrelay: info: GSmtp::ServerProtocol: rx<<: "."
emailrelay: info: GStore::NewFile: envelope file: out/emailrelay.49933.1673124642.2.envelope.new
emailrelay: info: GStore::FileDelivery::deliver: delivery: delivering out/emailrelay.49933.1673124642.2.envelope.local to [local]
emailrelay: warning: GSmtp::ProtocolMessageStore::process: message processing exception: delivery: cannot open envelope file: /home/stokito/spool/out/emailrelay.49933.1673124642.2.envelope.local
emailrelay: info: GSmtp::ServerSend: tx>>: "452 failed"

The error occurs right after the dot . is sent. I removed the --out-address-verifier local: and then I didn't get an error and the messaged was stored into /spool/out folder.

Here is a commad to start that I used:

emailrelay \
--log --verbose --debug \
--no-daemon \
--in-spool-dir /home/stokito/spool/in \
--in-port 10025 \
--in-domain stokito-serv \
--in-address-verifier allow: \
--in-filter deliver: \
--in-pop \
--in-pop-port 10110 \
--in-pop-auth /pam \
--in-pop-by-name \
--out-spool-dir /home/stokito/spool/out \
--out-port 10587 \
--out-domain stokito-serv \
--out-address-verifier local: \
--out-local-delivery-dir /home/stokito/spool/local

Graeme Walker - 2023-01-07

I wonder if the logging is a bit misleasing and it's just that the spool/local directory does not exist?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-01-07

No, okay, you remember I said it wasn't release quality...? I think it only works with --local-delivery-dir if the message has a local recipient :-< You don't have a local recipient in your test because the domain names don't match. You would need "--out-domain=disk.stokito.com".

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-01-07

I've checked in a quick fix for the problem, but even in its broken state I hope it gives you an idea of where the code is going. Thanks for giving it a go.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-01-14

I think I'm getting to the end of my winter coding assignment... The latest check-in to the V_2_5 branch has built-in filters to do routing and delivery, so almost a real MTA! There is some very brief documentation is in the NEWS file.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Ponomarev - 2023-01-29

Thank you
I pushed the v2.5 to the Ubuntu PPA https://code.launchpad.net/~stokito/+archive/ubuntu/emailrelay

So if anyone wants to test it on VPS you can install already compiled

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-08-11

status: open --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Graeme Walker - 2023-08-11

2.5 is now released, with more built-in filters, improved routing, client account selection, simplified dnsbl configuration, support for delivery and multiple configurations -- all as requested! Release notes are in the NEWS file.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Emailrelay as a Self hosting Email server on OpenWrt router

SMTP email proxy and relay server

Group

Searches

Help

#75 Emailrelay as a Self hosting Email server on OpenWrt router

Discussion