Re: [webmin-l] Virtualmin Professional

[Joe C. <jo...@sw...>]

Craig White wrote:
> On Thu, 2005-09-15 at 17:01 -0500, Joe Cooper wrote:

> I'm going to try to keep this a little more tightly defined as you have
> enough on your plate.
> 
> Much of my point involves 'real users' vs. 'virtual users' - Those with
> a shell and a home directory and those that don't have a shell or a home
> directory. The user db being /etc/passwd or sqldb or sasldb or ldap db

Ah, now we're getting to the heart of the matter, and one I can 
understand...though I suspect (I don't know yet who our customers will 
be, but this is my suspicion) that we're not targeting mass-mail servers 
like you're operating (though it sounds like a nice setup).  I'm gonna 
have to consult with our customers to find out if this is something 
they'd want.  There is no reason we couldn't implement it this 
way...it's just that having user accounts provides a lot of benefits and 
features that just aren't possible, or at least not without getting 
hopelessly complex in the implementation, without user accounts (whether 
those user accounts are stored in LDAP or /etc/passwd is orthogonal, and 
we'll be supporting LDAP more in the future).

> Sieve - If you were familiar with Horde/IMP/Ingo in it's current
> incarnation, you would understand. It's virtually out of the box set up
> so that users can maintain their own filters, set their own vacation
> replies, blacklists, whitelists, and 'fileinto' folders. Unlike
> procmail, user doesn't need to have a shell account at all.

Sounds very cool indeed.  I'm impressed by how far Horde/IMP/Ingo have come.

Virtualmin Professional sets things up in such a way that all of these 
things are possible, as well, at both the user and domain level. 
Usermin just doesn't provide an easy to understand way to configure 
them.  Auto-reply has gotten much easier in the latest release, but the 
rest of it definitely needs work.  It's very well hidden right now.  The 
use does have to have a shell account, however, but I don't view this as 
a negative--we need that shell account for other stuff anyway.  If 
having shell accounts is a problem, simply disabling all login types 
other than POP/IMAP/webmail brings the user account down to the level of 
a mailbox account in a system like you describe.  So, we can drop down 
to that level if we have users, but we can't climb up to having user 
account features if we don't have user accounts.  ;-)

> I used procmail for a few years and I liked it. Ingo (part of the Horde
> project) does interface with it and is probably the best example I have
> seen of making procmail user friendly. When I migrated to cyrus-imapd,
> sieve was part of the migration and it's simple enough - in fact, not as
> powerful as procmail but it has none of the overhead of procmail either.

I've not seen it in action, so I'll have to spend some time with Sieve 
to know.  I don't find procmail a resource problem on any of our test 
boxes, but admittedly our biggest test box only gets/send 20,000 or so 
messaages a day.  I think, again we're back to the question of "what is 
the system being used for".  And I don't think mass-hosting of email 
accounts is what Virtualmin Professional servers are going to be doing. 
   Virtualmin does have a mode of operation where it can deal 
exclusively with a single domains mail, plus private websites (i.e. 
~username style sites), but it's pretty much un-used by anyone except 
the company that sponsored that feature years ago.

It would actually be a pretty cool product to develop--a 
Webmin/Usermin/Mailmin(tm) bundle that sets all of the stuff up similar 
to what you've described with no user accounts and tuned for extreme 
performance.  We won't be developing it at Virtualmin, Inc. but maybe 
someone else will.  ;-)

> Cyrus - LDAP - virtual users, no shell account whatsoever. Mailstore is
> complete in it's own tree and not spread in user directories. Virtual
> users don't have home directories to maintain. Cyrus scales - with it's
> murder architecture, you can spread the load among servers. Cyrus has
> shared mailboxes, public mailboxes, quotas. Dovecot - for all it's
> features clearly isn't in the same league.

This again becomes a problem for Virtualmin.  We want the mail spools to 
be spread out in the user directories and owned by the user.  Disk 
quotas rely on this fact about Maildir to do their job.  Load is 
probably not our big concern with email in a virtual domain system.

> I have to believe that you have chosen dovecot for your pop/imap server
> because you really aren't familiar with cyrus-imapd (or ldap).

Hehehe...No comment.  (Other than: Dovecot supports deployment in a 
manner that looks a lot like Cyrus including LDAP users, if you really 
wanted to go that route...where Cyrus does not support deployment in a 
manner that looks like Dovecot.)

> I guess when I see so many large installations of universities and
> corporations using cyrus-imapd, horde/imp, it seems to be an indication
> of their assessment of the performance, scalability and feature sets
> that some people do want.

Certainly.  I just don't believe mail performance is the biggest concern 
on a Virtualmin server.  If it proves to be a problem (I've never heard 
reports of it being one), we'll definitely address it.

> Now if your target is the virtual hosting companies serving 15-30
> domains and 500-3000 users, perhaps these issues are less important.

No, we're targeting much larger servers than that, at least from a 
number of domains perspective, and we have it on servers much larger 
than that.  We expect some environments will have 500 or more (small) 
domains per server.  But I think you over-estimate the resource usage of 
Postfix+Dovecot+Procmail and under-estimate the resource usage of 
SpamAssassin+ClamAV (these two are where /all/ of our servers mail work 
is going on...Procmail is not even a blip on the radar...and we can't 
drop Spam and AV scanning and I don't know of equal products in this 
field that require fewer resources).  I'd be willing to wager that a 
Cyrus+Sieve server is no more than 5% more efficient than a 
Procmail+Dovecot server, when anti-virus and anti-spam measures are 
identical on each.  In short, I respectfully disagree with your 
assessment of where resource usage goes in a mail server--at least mail 
servers running in a virtual hosting environment like where Virtualmin 
is intended to be running.

> This of course, represents my own limited view and it's clear to me that
> you and Jamie have given this a lot of thought and have quite a lot more
> experience among you both than I have (by a very large margin). And of
> course, you can adapt over time anyway.

You clearly have a lot of good mail server operating experience, 
possibly more than Jamie or I, and I'm not going to discount it.  I only 
disagree with you on these few points because I believe we're targeting 
a different type of deployment than you've seen.  What is good for a 
dedicated high-load mail server may not be what is good for a shared 
multi-domain hosting environment with many admins (limited to their own 
domain, of course) and many users of widely varying knowledge levels.  I 
believe we've gone the right direction, but if we see a customers server 
taking a beating because of the way we've handled mail accounts, we'll 
fix it--and maybe the way to fix it will be to add a fully virtualized 
LDAP mailbox account type, or maybe it will be to remove procmail from 
the equation (but again, on our servers, SpamAssassin and ClamAV are 
where all of our CPU is going and there's nothing I'm aware of that we 
can do to remove that work).

> ps - if you would like to see a cyrus/sieve - horde/imp/sieve/(more)
> installation, I could create a login account for you at one of my
> clients systems.

I wouldn't turn down a chance to see one up and running nicely.  It's 
been at least three years since I've setup Horde for testing (and I 
ended up with OpenWebmail back then for my client deployment), so it 
would be a distraction that I can't afford to take on right now.  But 
I'd certainly like to see it...sounds like maybe they've gotten filter 
editing right, and we definitely need to improve filter editing in 
Usermin.  You didn't even know all of this stuff was possible with 
Usermin, so clearly it needs to be implemented in an easier to use way!

Thanks for the follow-up post.  As you've said, we are willing to adapt, 
and it may be that I have to revisit this issue in a month or six when a 
hosting provider with a very different workload shows up (maybe they 
host servers with twenty domains and 10,000 email accounts for each 
domain, in which case we'll probably need to rethink everything about 
email).

Regards,
Joe