From: Juergen N. <jue...@fu...> - 2012-09-19 17:12:49
Attachments:
signature.asc
|
Hello all, we have a SquirrelMail installation with around 5000 active users per day, ~14000 in three weeks, and switched to 1.4.22 and using imapproxy a few weeks ago. Everything runs peachy, so SquirrelMail has again proven to be capable and reliable. Big Thanks to the developers! Yet, in a few single cases, users have complained that the download of a large attachment failed due to the object being no longer present, according to SquirrelMail, and after clicking on the folder name, SquirrelMail showed the folder as empty. Only after logout and login again their messages were shown in the folder as before. This *seems* to correlate with the IMAP server (Dovecot) complaining, e.g., "Disconnected for inactivity in reading our output bytes=3588/19231114". The size of the attachment in two other cases was around 8 MB and around 11 MB, respectively. The imapproxy doesn't log anything related, apparently. As mentioned, this is really rare. We have had 3 users (that I know of) report that, one of them in 3 cases. The logs don't seem to show anything related except Dovecot as cited above. The users are not really happy with it, though; seeing their mails as lost looks like a nasty surprise. I'd really love to fix that or work around it. Any ideas? Regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |
From: Tomas K. <to...@us...> - 2012-09-22 07:40:52
|
Juergen Nickelsen-3 wrote: > > Hello all, > > ... > > Yet, in a few single cases, users have complained that the download of a > large attachment failed due to the object being no longer present, > according to SquirrelMail, and after clicking on the folder name, > SquirrelMail showed the folder as empty. Only after logout and login > again their messages were shown in the folder as before. > > This *seems* to correlate with the IMAP server (Dovecot) complaining, > e.g., "Disconnected for inactivity in reading our output > bytes=3588/19231114". > > The size of the attachment in two other cases was around 8 MB and around > 11 MB, respectively. > > The imapproxy doesn't log anything related, apparently. > > As mentioned, this is really rare. We have had 3 users (that I know of) > report that, one of them in 3 cases. The logs don't seem to show > anything related except Dovecot as cited above. > > The users are not really happy with it, though; seeing their mails as > lost looks like a nasty surprise. I'd really love to fix that or work > around it. > > Any ideas? > What's your PHP memory limit and are APC or commercial Zend extensions used in your setup? Do you log PHP errors? Maybe that "empty folder" error was caused by PHP error and your users saw unfinished page generation instead of normal SquirrelMail empty folder view. -- Tomas -- View this message in context: http://old.nabble.com/SquirrelMail-aborting-large-attachment-download--tp34453593p34465505.html Sent from the squirrelmail-users mailing list archive at Nabble.com. |
From: Juergen N. <jue...@fu...> - 2012-09-24 11:55:22
Attachments:
signature.asc
|
On 22.09.2012 09:40, Tomas Kuliavas wrote: > Juergen Nickelsen-3 wrote: [...] >> Yet, in a few single cases, users have complained that the download of a >> large attachment failed due to the object being no longer present, >> according to SquirrelMail, and after clicking on the folder name, >> SquirrelMail showed the folder as empty. Only after logout and login >> again their messages were shown in the folder as before. [...] > What's your PHP memory limit and are APC or commercial Zend extensions used > in your setup? 256 MB (we could increase that -- is that per Session or for all of them?); no commercial extensions, rather PHP as it comes out of the (Debian) box with a few common extensions: libapache2-mod-php5 5.3.3-7+squeeze14 php-db 1.7.13-2 php-mdb2 2.5.0b2-1 php-mdb2-driver-mysql 1.5.0b2-1 php-pear 5.3.3-7+squeeze14 php5-cli 5.3.3-7+squeeze14 php5-common 5.3.3-7+squeeze14 php5-curl 5.3.3-7+squeeze14 php5-gd 5.3.3-7+squeeze14 php5-ldap 5.3.3-7+squeeze14 php5-mysql 5.3.3-7+squeeze14 php5-suhosin 0.9.32.1-1 (I see that we don't use a few of them any more...) The Debian version is squeeze, as you can see, and the box is an HP ProLiant BL460c G6 blade with 8 cores and 32 GB RAM, so hardware resources are plenty. > Do you log PHP errors? Maybe that "empty folder" error was caused by PHP > error and your users saw unfinished page generation instead of normal > SquirrelMail empty folder view. Quite likely, and thanks for the hint! I see now that the PHP logging configuration was hosed when I moved the web server from one function account to the other. I will now get this sorted out and see what I can find in the error logs. If necessary I'll come back again. Thanks again! Regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |
From: Tomas K. <to...@us...> - 2012-09-24 16:04:09
|
2012.09.24 14:55 Juergen Nickelsen rašė: > On 22.09.2012 09:40, Tomas Kuliavas wrote: > >> Juergen Nickelsen-3 wrote: > [...] >>> Yet, in a few single cases, users have complained that the download of >>> a >>> large attachment failed due to the object being no longer present, >>> according to SquirrelMail, and after clicking on the folder name, >>> SquirrelMail showed the folder as empty. Only after logout and login >>> again their messages were shown in the folder as before. > [...] >> What's your PHP memory limit and are APC or commercial Zend extensions >> used >> in your setup? > > 256 MB (we could increase that -- is that per Session or for all of > them?); no commercial extensions, rather PHP as it comes out of the > (Debian) box with a few common extensions: > > libapache2-mod-php5 5.3.3-7+squeeze14 > php-db 1.7.13-2 > php-mdb2 2.5.0b2-1 > php-mdb2-driver-mysql 1.5.0b2-1 > php-pear 5.3.3-7+squeeze14 > php5-cli 5.3.3-7+squeeze14 > php5-common 5.3.3-7+squeeze14 > php5-curl 5.3.3-7+squeeze14 > php5-gd 5.3.3-7+squeeze14 > php5-ldap 5.3.3-7+squeeze14 > php5-mysql 5.3.3-7+squeeze14 > php5-suhosin 0.9.32.1-1 PHP APC and Zend Accelerator (or some other component from Zend Platform package) extensions reduce peak memory usage. PHP APC is in main Debian package repo. In PHP 5.2.0+ it is possible to check memory usage with plugin attached to some standard SquirrelMail hooks. 256 MB is high enough and it should be difficult to hit that limit, but peak usage can be affected by memory usage in previous page load. Mailbox display and message view display can trigger higher usage under some conditions. Maybe others have different experience. For me reported peak memory usage differences made PHP APC a must in webmail. Do you have server side sorting enabled and server side threading disabled in SquirrelMail config? Do you use gpg or uudecode plugins? Do those large emails have more text in plain text or html message parts? If possible, check logs for suhosin related errors or try running webmail with suhosin set to simulation mode. -- Tomas |
From: michael c. <mic...@gm...> - 2012-09-24 12:30:31
|
On Mon, September 24, 2012 12:55 pm, Juergen Nickelsen wrote: > On 22.09.2012 09:40, Tomas Kuliavas wrote: > >> Juergen Nickelsen-3 wrote: > [...] >>> Yet, in a few single cases, users have complained that the download of >>> a >>> large attachment failed due to the object being no longer present, >>> according to SquirrelMail, and after clicking on the folder name, >>> SquirrelMail showed the folder as empty. Only after logout and login >>> again their messages were shown in the folder as before. > [...] >> What's your PHP memory limit and are APC or commercial Zend extensions >> used >> in your setup? > > 256 MB (we could increase that -- is that per Session or for all of > them?); no commercial extensions, rather PHP as it comes out of the > (Debian) box with a few common extensions: I thought the PHP limit was per message. as a general list question are most people's mail servers attached directly to the internet ? mick -- keyID: 0x4BFEBB31 |
From: Juergen N. <jue...@fu...> - 2012-09-24 12:44:32
Attachments:
signature.asc
|
On 24.09.2012 14:30, michael crane wrote: [PHP memory limit] >> 256 MB (we could increase that -- is that per Session or for all of >> them?); no commercial extensions, rather PHP as it comes out of the >> (Debian) box with a few common extensions: > I thought the PHP limit was per message. You mean per script run, I assume? The manual says[1] "the maximum amount of memory in bytes that a script is allowed to allocate." This would AIUI mean per script run, but I don't know if this is somehow changed by running as mod_php in the Apache process. [1] http://php.net/manual/en/ini.core.php > as a general list question are most people's mail servers attached > directly to the internet ? Define "attached directly to the internet". The one I run for our university is behind more than one firewall, of course, but reachable from the Internet. Regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |
From: michael c. <mic...@gm...> - 2012-09-24 13:22:12
|
On Mon, September 24, 2012 1:44 pm, Juergen Nickelsen wrote: > On 24.09.2012 14:30, michael crane wrote: > > [PHP memory limit] >>> 256 MB (we could increase that -- is that per Session or for all of >>> them?); no commercial extensions, rather PHP as it comes out of the >>> (Debian) box with a few common extensions: >> I thought the PHP limit was per message. > > You mean per script run, I assume? > > The manual says[1] "the maximum amount of memory in bytes that a script > is allowed to allocate." This would AIUI mean per script run, but I > don't know if this is somehow changed by running as mod_php in the > Apache process. > > [1] http://php.net/manual/en/ini.core.php > >> as a general list question are most people's mail servers attached >> directly to the internet ? > > Define "attached directly to the internet". The one I run for our > university is behind more than one firewall, of course, but reachable > from the Internet. I only use on my home network, it seemed problematic to get my own IPaddress and advertise my mail server when I could use gmail to hold the mail. -- keyID: 0x4BFEBB31 |
From: Juergen N. <jue...@fu...> - 2012-10-09 10:38:17
Attachments:
smime.p7s
|
[TLDR: taking out imapproxy resolved the issue; we'll live with that for now] Hello, three weeks ago, I wrote this: > [...] in a few single cases, users have complained that the download of a > large attachment failed due to the object being no longer present, > according to SquirrelMail, and after clicking on the folder name, > SquirrelMail showed the folder as empty. Only after logout and login > again their messages were shown in the folder as before. > > This *seems* to correlate with the IMAP server (Dovecot) complaining, > e.g., "Disconnected for inactivity in reading our output > bytes=3588/19231114". First, thanks for the comments and detail questions regarding this issue! Please excuse me for not following up earlier -- you know how it is sometimes. In between, summer break nearly over, user complaints about this problem became more frequent. As we more and more suspected the imapproxy to be part of it, we configured it out of the loop a week ago, and lo, user complaints stopped. Now a colleague of mine succeeded to reproduce the problem reliably on a test system, where the imapproxy is still in use, by cancelling the download of a large attachment. Following that, SquirrelMail shows all mail folders as empty -- just as our users had reported it -- until the cached connection in the imapproxy expires. This behaviour is not seen when the imapproxy is not used. Actually, the IMAP chain is even longer: SquirrelMail <--> imapproxy <--> Perdition <--> Dovecot* The user mailboxes are distributed over a number of Dovecot servers for performance reasons, and Perdition relays the IMAP connection to the responsible server. I looked around the imapproxy source if there were any additional tuning or scaling parameters and didn't find any; unfortunately I will not be able short-term to dig deeper into it to find out if this is a problem with imapproxy in general or perhaps only in interaction with Perdition. While having imapproxy taking load off Perdition would be nice, we can do without for now. If anyone has a specific idea what can be done about the problem, I'd love to hear about it, but otherwise I'll leave it at that for the moment. Thanks again for your effort to help -- I appreciate that a lot. Best regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |
From: Tomas K. <to...@us...> - 2012-10-09 16:31:35
|
Juergen Nickelsen-3 wrote: > > The user mailboxes are distributed over a number of Dovecot servers for > performance reasons, and Perdition relays the IMAP connection to the > responsible server. > ... > While having imapproxy taking load off Perdition would be nice, we can > do without for now. > If you use Perdition only for directing users to proper Dovecot instance, you can as well take Perdition out of the picture. SquirrelMail can map users to different servers with PHP function linked to IMAP server settings. Core scripts have sample function only for YP/NIS, but maybe your developers might create similar functions for backend used by Perdition. -- Tomas -- View this message in context: http://old.nabble.com/SquirrelMail-aborting-large-attachment-download--tp34453593p34533381.html Sent from the squirrelmail-users mailing list archive at Nabble.com. |
From: Juergen N. <jue...@fu...> - 2012-10-11 14:54:29
Attachments:
smime.p7s
|
On 09.10.2012 18:31, Tomas Kuliavas wrote: >> While having imapproxy taking load off Perdition would be nice, we can >> do without for now. >> > If you use Perdition only for directing users to proper Dovecot instance, > you can as well take Perdition out of the picture. No, because SquirrelMail is not the only IMAP client; actually we have more users with other IMAP clients than webmail users. Regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |
From: Paul L. <pa...@sq...> - 2012-10-09 16:32:43
|
On Tue, Oct 9, 2012 at 3:38 AM, Juergen Nickelsen <jue...@fu...> wrote: > [TLDR: taking out imapproxy resolved the issue; we'll live with that for > now] > > Hello, > > three weeks ago, I wrote this: > >> [...] in a few single cases, users have complained that the download of a >> large attachment failed due to the object being no longer present, >> according to SquirrelMail, and after clicking on the folder name, >> SquirrelMail showed the folder as empty. Only after logout and login >> again their messages were shown in the folder as before. >> >> This *seems* to correlate with the IMAP server (Dovecot) complaining, >> e.g., "Disconnected for inactivity in reading our output >> bytes=3588/19231114". > > First, thanks for the comments and detail questions regarding this > issue! Please excuse me for not following up earlier -- you know how it > is sometimes. > > In between, summer break nearly over, user complaints about this problem > became more frequent. As we more and more suspected the imapproxy to be > part of it, we configured it out of the loop a week ago, and lo, user > complaints stopped. > > Now a colleague of mine succeeded to reproduce the problem reliably on a > test system, where the imapproxy is still in use, by cancelling the > download of a large attachment. Following that, SquirrelMail shows all > mail folders as empty -- just as our users had reported it -- until the > cached connection in the imapproxy expires. This behaviour is not seen > when the imapproxy is not used. > > Actually, the IMAP chain is even longer: > > SquirrelMail <--> imapproxy <--> Perdition <--> Dovecot* > > The user mailboxes are distributed over a number of Dovecot servers for > performance reasons, and Perdition relays the IMAP connection to the > responsible server. Why not Dovecot Director? If you have testing resources, you could see if removing Perdition also takes care of the problem. > I looked around the imapproxy source if there were any additional tuning > or scaling parameters and didn't find any; unfortunately I will not be You can watch log messages from IMAP Proxy and see if it thinks its cached connection is as normal. If so, I'd suspect that it's as simple as Dovecot got tired of waiting around and closed that connection out from under the proxy(ies) (which the Dovecot log message you found seems to support). When you reproduced it in testing, was that Dovecot log message always there? If this is the case, I'd suspect that the problem is more on the Dovecot side, possibly because one of the layers above it reads and holds/serves the attachment but doesn't close the connection until the browser is finished. There may be some timeout settings you can tweak, and you might need to reconsider your attachment size limitations. > able short-term to dig deeper into it to find out if this is a problem > with imapproxy in general or perhaps only in interaction with Perdition. Well keep us updated and thanks for the follow-up. > While having imapproxy taking load off Perdition would be nice, we can > do without for now. If anyone has a specific idea what can be done about > the problem, I'd love to hear about it, but otherwise I'll leave it at > that for the moment. > > Thanks again for your effort to help -- I appreciate that a lot. -- Paul Lesniewski SquirrelMail Team Please support Open Source Software by donating to SquirrelMail! http://squirrelmail.org/donate_paul_lesniewski.php |
From: Juergen N. <jue...@fu...> - 2012-10-11 15:33:02
Attachments:
smime.p7s
|
On 09.10.2012 18:32, Paul Lesniewski wrote: >> The user mailboxes are distributed over a number of Dovecot servers for >> performance reasons, and Perdition relays the IMAP connection to the >> responsible server. > > Why not Dovecot Director? If you have testing resources, you could > see if removing Perdition also takes care of the problem. That is not something I implemented; While I cannot recall the exact reason, I remember Dovecot Director was considered, but deliberately not used. Unfortunately I will not be able to investigate the issue further short-term; as we have an acceptable workaround at the moment, other tasks are much more urgent now. Thanks again! Regards, Juergen. -- <Jue...@fu...> Tel. +49.30.838-50740 Zentraleinrichtung fuer Datenverarbeitung, Central Systems (Unix) Freie Universitaet Berlin, Fabeckstrasse 32, 14195 Berlin, DE |