From: Thomas Eckardt/e. <Tho...@th...> - 2008-08-23 06:53:15
|
Max, every thread makes a MySQL connection for every list/cache and does a connection check every two minutes to hold them up. We are accessing the list/caches simply like any %HASH in Perl, and after start ASSP does 'not know' if the list/cache is native in RAM or if it is tieed to a database. For this reason every connection must be available at any time! Yes, if you are running 15 threads (SMTP-workers) and you are using DB for all lists/caches you'll have 18*20 = 360 MySQL connection opened for ASSP - no way out! That is the point where some MySQL servers gives up. Try to tune your MySQL configuration to handle that many connections. I've made some tests for that: MySQL - 5.0.x is doing it with innodb, but 5.1.x , 4.x.x have trubble to handle that large amount of connection! How ever MSSQL (2000/2005 - including the free Express installation) , oracle 9.204 and higher and DB2 does not have that problem(ADO,ODBC or native driver)! Thomas Maxwell Rathbone <mra...@sa...> Gesendet von: ass...@li... 22.08.2008 20:49 Bitte antworten an ASSP development mailing list <ass...@li...> An ASSP development mailing list <ass...@li...> Kopie Thema Re: [Assp-test] Antwort: Re: Antwort: Re: Antwort: SMTP Timeout Problem Understand that is a serious limitation of the implementation of mySQL then. What I'm seeing on my side is either that ASSP is not utilizing the persistent connections, or it is not properly closing the mySQL connection. This is resulting in hundreds of sleeping connections on the mySQL Server. Just now, our website was down as ASSP utilized every available mySQL connection on the mySQL Server. I just doubled the allowed connections on the mySQL server, and set the max_user_connections to 25. This will start throwing connection errors to ASSP unless it is properly utilizing the persistent connection. thank you Max Thomas Eckardt/eck wrote: >Is there any way to turn off persistent mySQL database connections? No! Thomas Maxwell Rathbone <mra...@sa...> Gesendet von: ass...@li... 22.08.2008 18:22 Bitte antworten an ASSP development mailing list <ass...@li...> An ASSP development mailing list <ass...@li...> Kopie Thema Re: [Assp-test] Antwort: Re: Antwort: SMTP Timeout Problem Thanks for the tips. I adjusted the workers to 15, and left 5 reserved for outgoing emails. This alone seemed to help pretty considerably. I also installed a local caching nameserver and changed the resolv.conf to point to it. I was hoping that a local DNS server might also speed up things a little bit. I was able to get the mySQL back-end going(off-server mySQL Daemon). I do have one concern however regarding the use of persistent connections. I'm noticing a TON of sleeping assp connections on the mySQL server. While we do have a very beefy mySQL server, it is a shared environment. So if for some reason assp keeps opening connections up and leaving them sleeping, it could conceivably bring down the mySQL server by preventing other services from making a connection. Is there any way to turn off persistent mySQL database connections? I had a piece of code that did persistent connections for SpamAssassin and ended up having to remove it because it had a similar problem. I am keeping the ASSP on the mail server itself, mainly due to this being such a high-end system. I'd guess I'm part of a very small minority on this however, and would otherwise normally follow the stated recommendation. Thanks for the tips.. will keep an eye on it today and see how it fares. thanks Max Thomas Eckardt/eck wrote: Hello Maxwell, try to increase the number of SMTP-Workers and to reserve 1-3 for outgoing emails (server setup). Depending on the systems hardware configuration it should not be a problem to run up to 30 SMTP-Workers. If ASSP gets a new connection request, it is looking to find the best worker for that. Best is a sleeping worker, but if they all have something to do, the MainThread will interrupt the worker with the lowest number of opened connections. So, if your system gets often much more new connection requests than running workers at the same time, the responds time will increase dramaticly - having some sleeping workers for those peeks will help! And for all to know! Just keep in mind ! I do not recommend to run ASSP 2 and the receiving MTA on the same (heavy load) system. ASSP 1 uses only one thread, so if you have a multicore system the MTA will run on the other core, if ASSP is going to use one core on 100% there will be no problem. ASSP 2 is multithreaded - if (for example) you have 4 cores and each of 4 SMTP-workers uses 100% for some reasons, the MTA will not get (or less) CPU-time! And ASSP will not get a fast answer from the MTA. The same belongs to the DB if used. If the DB-daemon is running on the same system and ASSP 2 uses 100% (which can be normal for some seconds) of all cores, ASSP will not get a fast responds from the DB-engine. I recommend to keep the MTA away from the ASSP 2 -system and to reserve one core for the DB-daemon (CPU affinity), if it is used on the same heavy load system! Most of the checks in ASSP are using DNS - if you are use a slow DNS-server - ASSP will be slow! It should never take more than two seconds to get a responds from your DNS-server. If a system has not the right hardware (and software) configuration (related to the number of connection requests and the configured ASSP checks) or it has more to do than handling ASSP, it could be better to use a singlethread version of ASSP! And to answer the question "Do I have a heavy load system?" - "heavy load" IMHO, is more than one new connection per second (round more than 100.000 a day)! Handling a number of 500.000 (or even more) connections a day should be no problem with ASSP 2. A well designed ASSP-system should show in the 'connection list' an average processtime per email between 3 an 5 seconds (even less is better) - connection that are running in to SMTP-Timeout are not counted! This is only the top of the surface, for this reason, we (Fritz and I) call ASSP 2 - "ASSP professional" - it could take some time to find the right ASSP-configuration, and some times there is more to know and to do, than clicking some buttons! If you are new to ASSP or you do not know much about IP,DNS,SMTP and system design - first try ASSP 1.x.x to get some experiences. Thomas Maxwell Rathbone <mra...@sa...> Gesendet von: ass...@li... 21.08.2008 14:17 Bitte antworten an ASSP development mailing list <ass...@li...> An ASSP development mailing list <ass...@li...> Kopie Thema Re: [Assp-test] Antwort: SMTP Timeout Problem I tried completely turning logging off and it does not appear to be helping the situation. :( I'm seeing upwards of 9 seconds+ response time before ASSP provides the 220. I'm going to leave logging off and immediately start working on moving to a mySQL backend with hopes that it will help. Really, it's the initial delay more than anything that is causing so much trouble for me. thanks Max Thomas Eckardt/eck wrote: At this time there is a 'unsightly design' in ASSP 2, which can cause this delay on heavy load systems, if logging is expanded! The MainThread handles the initial connection requests and swap them in to the workers, but in addition to this 'work' the MainThread is doing the logging (STDOUT,maillog,syslog) for all Threads. So disable most of the logging options should help to solve this problem (connection log, session log, worker log). I hope I can change this! Memory: ASSP 2 uses a totaly different way to access the lists and caches (whitelist,redlist,spamdb,spamdb.helo,delaydb,ldaplist,penaltybox). ASSP 1 uses orderedtie, which need less memory. ASSP 2 has to tie all lists and caches to all threads and for this reason all lists and caches are loaded in to memory! If you make a summary of the filesize of all lists and caches * 1.5 * (workernumber +3) you'll get the approximatly needed memory. We've made many test to find a better way to handle the memory. The best is to use a database for all lists and caches to reduce the memory. This will need some more memory for code, but much less memory for the data! For example: a windows system with 10 SMTP-workers on perl 5.8.8 build 822 and more than 100.000 connection requests a day and a replicated database needs between 400 and 450 MB RAM for ASSP and 70MB for the mysql daemon! Be careful using the OCR-Plugin. The PDF-to-text option is no problem, but using ImageMagic and tesseract on large images, could cause in a 'system out of memory' error. All compressed images have to be decompressed by ImageMagic to process them with tesseract. But a 2MB compressed jpeg could be 30MB decompressed. Doing OCR with tesseract on such a large image will freeze your system. Use 'ASSP_OCRocrmaxsize' to prevent that! monitoring ASSP: Try to reduce the monitoring requests (for example nagios check on port 25 using telnet) to a minimum. Any connection request to ASSP cause a memory reservation insite ASSP for this new connection. So monitoring ASSP one time every second on port 25, cause 86.400 reservation requests a day! Thomas Maxwell Rathbone <mra...@sa...> Gesendet von: ass...@li... 20.08.2008 17:57 Bitte antworten an ASSP development mailing list <ass...@li...> An ASSP development mailing list <ass...@li...> Kopie Thema [Assp-test] SMTP Timeout Problem Description: Long delay, or no answer at all when telnet'ing to port 25 (ASSP). Performing a telnet to port 125 has the mail server answer immediately. This is a very troublesome problem as it is not only causing our monitoring system to go crazy, but it is causing customers to get upset as they cannot connect to the mail server to send mail. It is worth noting that the mail server itself is working great and answers to port 125 immediately. It is ASSP that is causing the delay. At this point in time, the assp.pl process is utilizing 55.7% of the systems total memory(2GB) and that appears to be rising the longer it is running. This may be the cause of the delay, or possibly a symptom of the greater problem. ------------------------------------------------------------------------------------------- Server Information: OS & Perl Information: [root@mail assp]# cat /etc/redhat-release CentOS release 5.2 (Final) [root@mail assp]# rpm -qa | grep kernel- kernel-2.6.18-53.1.21.el5 [root@mail assp]# rpm -qa | grep perl-5 perl-5.8.8-10.el5_2.3 *snippet of log* Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Aug 20 11:52:46 mail assp[22178]: [Worker_2] info: Worker_2 is interrupted to get new connection Aug 20 11:52:46 mail assp[22178]: [Worker_2] Info: Worker_2 got connection from MainThread Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread freed by interrupted Worker_2 Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Aug 20 11:52:46 mail assp[22178]: [Worker_4] Info: Worker_4 is interrupted to get new connection Aug 20 11:52:46 mail assp[22178]: [Worker_4] Info: Worker_4 got connection from MainThread Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread freed by interrupted Worker_4 Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Aug 20 11:52:46 mail assp[22178]: [Worker_4] Info: Worker_4 is interrupted to get new connection Aug 20 11:52:46 mail assp[22178]: 47560-04105 [Worker_5] 119.112.90.59 <wgk...@ma...> to: co...@gr... recipient delayed: co...@gr... Aug 20 11:52:46 mail assp[22178]: 47560-04105 [Worker_5] [SMTP Error] 451 4.7.1 Please try again later Aug 20 11:52:46 mail assp[22178]: [Worker_5] Eb...@md... matches @mdrg.com in LocalAddresses_Flat Aug 20 11:52:46 mail assp[22178]: [Worker_4] Info: Worker_4 got connection from MainThread Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread freed by interrupted Worker_4 Aug 20 11:52:46 mail assp[22178]: 47564-07162 [Worker_1] 195.18.33.8 <omc...@br...> to: ba...@gr... recipient delayed: ba...@gr... Aug 20 11:52:46 mail assp[22178]: 47564-07162 [Worker_1] [SMTP Error] 451 4.7.1 Please try again later Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Aug 20 11:52:46 mail assp[22178]: [Worker_2] Info: Worker_2 is interrupted to get new connection Aug 20 11:52:46 mail assp[22178]: [Worker_2] Info: Worker_2 got connection from MainThread Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread freed by interrupted Worker_2 Aug 20 11:52:46 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Aug 20 11:52:46 mail assp[22178]: 47564-04711 [Worker_3] 201.25.88.190 <nic...@co...> to: o.m...@pm... recipient delayed: o.m...@pm... Aug 20 11:52:46 mail assp[22178]: 47564-04711 [Worker_3] [SMTP Error] 451 4.7.1 Please try again later Aug 20 11:52:46 mail assp[22178]: [Worker_2] Ws...@me... matches @mega-properties.com in LocalAddresses_Flat Aug 20 11:52:47 mail assp[22178]: 47566-10548 [Worker_2] 66.227.102.59 <Loo...@ho...> to: ws...@me... recipient delayed: ws...@me... Aug 20 11:52:47 mail assp[22178]: 47566-10548 [Worker_2] [SMTP Error] 451 4.7.1 Please try again later Aug 20 11:52:47 mail assp[22178]: [Worker_5] eb...@md... matches @mdrg.com in LocalAddresses_Flat Aug 20 11:52:47 mail assp[22178]: [Worker_1] Info: Worker_1 is interrupted to get new connection Aug 20 11:52:47 mail assp[22178]: [Worker_1] Info: Worker_1 got connection from MainThread Aug 20 11:52:47 mail assp[22178]: [Main_Thread] Info: Main_Thread freed by interrupted Worker_1 Aug 20 11:52:47 mail assp[22178]: [Main_Thread] Info: Main_Thread got connection request Max ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* |