#3692 Webmin Swap usage problem
Hello! I have a problem with webmin, I hope someone has a solution. I have a Fedora 10 server which basically does everything (routing, e-mail, DNS, FTP, HTTP, etc). I have been using Webmin for years now, and never had a problem... until now. My e-mail user accounts (Postfix + Unix accounts) are getting larger and larger, some of them over 8 - 10 GB. Everything worked fine, until one day one of my "special users" (I have some users who have permission in webmin to check e-mail accounts, view mail, etc) entered a 10 GB e-mail account and searched for a particular e-mail. That's when the server went berserk, started killing processes, I couldn't even ssh, telnet, etc. After some research I found out that when searching inside a huge mailbox, webmin was suddenly filling up all swap space (even though swap space was nearly unused: 3 MB out of 1024 MB). Is this a known bug, or is there some way to disable swap usage/memory usage when searching e-mail using webmin? I'm the latest stable version of webmin, 1.510-1 .
This sounds like a bug in the parsing of a large mail file.
Some questions :
1. What format is this mailbox in? A single mbox-format file, or a Maildir-format directory?
2. Has this large mailbox ever been searched before?
Thank you for your reply, jcameron.
1. It's a Maildir-format directory (with new, cur, .Sent, .Trash, .Drafts, etc folders)
2. Last time this user's mail was searched was a few weeks ago and it was smaller (6 GB in size)
There are some e-mail files in that Maildir, over 35 MB...
Did you happen to check which processes were using up a lot of RAM when you did this search? The 'top' command will tell you.
Searchign a large mailbox may be slow, but it shouldn't use up much RAM. However, it may generate lots of disk IO load..
As I said, once the search started, I couldn't even login let alone run commands. The server just stopped responding altogether and the only thing I could do was restart it manually (forced shutdown using the power button). After that, when checking /var/log/messages there were a lof of "Out of swap space" and "Process killed because of lack of swap space" :
kernel: Free swap = 0kB
kernel: Total swap = 1044184kB
kernel: 524235 pages RAM
kernel: 14454 pages reserved
kernel: 10418 pages shared
kernel: 500837 pages non-shared
kernel: Out of memory: kill process 2483 (httpd) score 95035 or a child
kernel: Killed process 2483 (httpd)
kernel: Out of memory: kill process 3803 (httpd) score 94769 or a child
kernel: Killed process 8109 (spamc)
kernel: Killed process 8181 (sendmail.postfix)
....
and so on...
If you dare, you could SSH in first, run top, then try searching mail. If you see some process using up a lot of RAM, you can kill it from within top, using the k key.
I definitely can't do this... There are hundreds of e-mail accounts and high traffic websites hosted on this server. If it goes offline again, someone is gonna get it... namely me :)
Oh well .. that will make this hard to debug then :-(
Well, today I had to search a 3 GB mailbox and managed to see which process was eating up a lot of memory. It's the miniserv.pl process. It eats all the available RAM then it starts eating the available swap... when all memory available for cache is gone... things go awry!
Here is a screenshot: http://img534.imageshack.us/img534/2076/miniservpl.png
Hope it helps... I really don't understand why it needs to cache all the emails just for something as simple as a search... :)
What kind of search did you do exactly? Was it just for a word in the subject or from address, or was it a search of the mail contents?
I did an Advanced Search... where any header ... contains ... "bla bla bla"
If you do a search just on the subject , does the same problem happen?
Yeah. I just did a search on the header "Subject". Again the memory is gradually occupied.
http://img197.imageshack.us/img197/2922/miniservpl2.png
I think to debug this further I would need to actually login to your system myself.
I have several Maildir-format mailboxes which are several GB in size, and I don't see this on my systems.
One more question though - which Webmin version do you have there? If it isn't 1.510, you should upgrade..
Yeah, I have the latest version, 1.510.
Oh well, I'll just use the CLI to search from now on... it's not a problem for me, but for some of my users... it will be a problem.