Throughput and ClientAccountSelector
SMTP email proxy and relay server
Brought to you by:
graeme_walker
Menu
▾
▴
#116 Throughput and ClientAccountSelector
Hi, we've recently introduced emailrelay to sit between our aging email-broadcast system and SendGrid and after a few hours getting it set up it has so far processed over 75,000 emails and counting.
Our biggest issue is one of throughput and it may not be on the emailrelay side. We're seeing between 60 and 120 messages enter the spool folder per minute, they're rapidly processed and the spool folder sits empty for a few seconds or so before another 60+ enter. This sounds pretty quick but we're dealing with mailings of over 15k recipients, so it can take upwards of an hour at this rate. Does this level of throughput sound about right? Or would you expect emailrelay to process far more than this and the issue potentially lie on our broadcast system?
My second question relates to ClientAccountSelector. We need different SendGrid credentials per sending domain so we have a filter setting the ClientAccountSelector based on the sending domain. I worry this might be part of the problem and wonder whether I can use the client ip as the ClientAccountSelector somehow without the need for a filter script? Our email-broadcast system essentially connects on a distinct ip per sending domain already, so if it were possible to use this as a ClientAccountSelector in the auth file we could skip the filter entirely.
Cheers,
Maxx
I can't really comment on the throughput you are seeing because there are so many variables.
One thing to bear in mind is that emailrelay runs as a single process and (mostly) using a single thread. It works asynchronously, so multiple connections are handled concurrently, but even so, for high performance you should try running multiple emailrelay processes sharing the same spool directory -- typically one server and multiple forwarding agents. If you can have multiple servers so much the better, perhaps by having each server listening on its own network interface.
The obvious answer to your second question is no, the remote client's IP address cannot be used as a ClientAccountSelector without a using filter. But if you are prepared to make code changes then a quick hack would be to re-write GFilters::CopyFilter::run() in src/gfilters/gcopyfilter.cpp -- read the envelope from file, copy from the "from" field to "client_account_selector", and write it back to file. So then "--filter=copy:" would do what you want without the overhead of calling an external script.
Please let me know what o/s you are using, because the answer to better performance is always better measurement and in that that regard I can only really help if you are running on Linux.
Thanks for the quick response and I appreciate my primary question was rather nebulous. The one server/multiple forwarding agents setup sounds achievable. I suppose I could benchmark this performance by simply operating our current setup as a server without forwarding. I could then time the inbound spooling.
I'll certainly give that code change a try, thanks for highlighting where I'd need to look.
Unfortunately we're running all this on a Windows 2022 Server. Have you any non-specific, top-level areas I should get my hardware folk to concentrate on?
Thanks again.
On reflection, modifying one of the built-in filters will given you a good idea of any overhead from running a script, but to avoid reading and writing the envelope file a more direct approach would be to fill in the client-account-selector at the time the envelope is first created. That is in GStore::NewFile::prepare() in src/gstore/gnewfile.cpp; add a line m_env.client_account_selector = peer_socket_address where it says 'save the envelope'.
On Windows I would be interested to know whether you are triggering the multi-threaded event loop. That happens when the number of connections and associated handles goes above the Win32 API limit of 64, and it introduces some extra inter-thread signaling and context switching. The Windows Task Manager will show you the number of threads and handles: select the 'details' tab and right-click on a column heading and then 'select columns'. If you discount the numbers at startup then you should see the number of threads increasing by one as the number of handles goes over a multiple of 64. The thread count will go up as needed, but it will never decrease, so a burst of activity might have a negative impact on throughput thereafter.
While you're there, use Task Manager to look at the CPU usage.
You can also use "netstat -n -o" to monitor the number of simultaneous connections in real-time, or perhaps do some log analysis to get a historical view.
Increasing the number of emailrelay processes is definitely worth a try. If having one server per network interface doesn't make sense in your setup then there might be some cleverer way of doing network load balancing that your local experts could advise on.
Also: check are that the spool directory and the log file are on local disks; try switching verbose logging off; make sure that you are not logging to the Windows Event Log.