pmilter-users Mailing List for PMilter: Pure-Perl Milter (Page 2)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Call out Gouranga be happy!!!
Gouranga Gouranga Gouranga ....
That which brings the highest happiness!!

Hi all,

I'm working on my first milter.  I chose to do it in Perl as I am vastly
more familiar with it than I am with C.  I justified any performance
questions I had with the fact that I currently use spamassassin which is all
Perl.  

So anyways, I wanted to give a heads up as to what I was working on to get
some feedback and to see if anyone was interested in a similiar solution.

First off, let me explain my mail environment.  I work for a site that has
100 - 120 current internet mail users.  All internal accounts are stored on
an old exchange server.  We have long been using sendmail in front of that
running on Debian to protect the exchange server from the internet and vice
versa.

Our company has been around a long time.  Thus, we have a large number of
users who have gone.  Some of these users are our largest spam recipients.
After running into problems with Exchange bogging down dealing with bounce
backs to non-existant email accounts, we implemented a user list on our
sendmail server.  Exchange (or so the NT administrator tells me) does not
have the capability to block an email if it is being sent to a non-existent
user.  Performing this blocking on the sendmail server cut down on email
traffic by two thirds.  The NT administrator and IT Director were both
relieved because they would spend hours per day cleaning out the NT queue
and this completely resolved that issue.

Now, sendmail is currently rejecting these emails on a per recipient basis.
In other words, it just strips off the invalid users and sends the mail on
if there are still valid users on the list.  Thus, it did nothing to
actually cut down on spam that valid users were receiving.

However, after seeing these drastic results and eyeballing recipient lists
on spam, we know that there is a good chance that if we just rejected
outright any mail that had an invalid user on it, we probably would reduce
greatly the amount of spam our valid users see.

I am working on a milter that accomplishes this.  It also will use the
whitelist that I have from spamassassin to look for recipients that we want
mail from to go through.  Thus, we will not run into problems like mailing
list email being rejected just because some old user never unsubscribed his
address before leaving the company.  It will however strip the invalid users
from a mail as sendmail would based on my previously described solution.
The user list and whitelist are both stored in hash files.

This is a pretty specific solution to the particulars of my environment and
I do not know how useful it would be to others or if someone else has
already done something like this.  Anyways, I would like your feedback in
any case.

Joe

On Thursday 05 August 2004 04:28, you wrote:
> On Wed, 4 Aug 2004, Todd Vierling wrote:
> > BTW, this doesn't preclude possibly allowing Net::Server in a
> > non-exclusive "latch-in" capacity in the near future.  I'm going to
> > add a RFE to SourceForge tracking this.
>
> So it seems Net::Server has become much more flexible since the last
> time I played with it -- when I looked at using it for pmilter, it
> was 0.84, updated back in 2002.  Net::Server 0.87
> (Net::Server::Proto) is now working correctly with arbitrary fds in
> my casual testing.
>
> This is SourceForge RFE 1003702, and it may well be implemented in
> 0.95 or 0.96 as a replacement for the dispatcher framework.  Thanks
> for the suggestion.

Thanks for accepting the suggestion :)

[CLOSED]

-- 
Andrzej [en:Andrew] Adam Filip an...@pr... an...@xl...
Home Page http://anfi.homeunix.net/ [ PageRank 6 ]

On Wed, 4 Aug 2004, Todd Vierling wrote:

> BTW, this doesn't preclude possibly allowing Net::Server in a non-exclusive
> "latch-in" capacity in the near future.  I'm going to add a RFE to
> SourceForge tracking this.

So it seems Net::Server has become much more flexible since the last time I
played with it -- when I looked at using it for pmilter, it was 0.84,
updated back in 2002.  Net::Server 0.87 (Net::Server::Proto) is now working
correctly with arbitrary fds in my casual testing.

This is SourceForge RFE 1003702, and it may well be implemented in 0.95 or
0.96 as a replacement for the dispatcher framework.  Thanks for the
suggestion.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

On Wed, 4 Aug 2004, Todd Vierling wrote:

> > Have you considered making PMilter "Net::Server friendly" ?
>
> Net::Server doesn't support IPv6 [yet], and its UNIX (filesystem) socket
> management leaves quite a bit to be desired.  If Net::Server were more
> *programmer*-friendly -- i.e. you could hand it an open listen socket of
> your own choosing -- then it wouldn't be so bad in this respect.  Maybe in
> the future....

BTW, this doesn't preclude possibly allowing Net::Server in a non-exclusive
"latch-in" capacity in the near future.  I'm going to add a RFE to
SourceForge tracking this.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

On Wed, 4 Aug 2004, Andrzej Filip wrote:

> Have you considered making PMilter "Net::Server friendly" ?

Net::Server doesn't support IPv6 [yet], and its UNIX (filesystem) socket
management leaves quite a bit to be desired.  If Net::Server were more
*programmer*-friendly -- i.e. you could hand it an open listen socket of
your own choosing -- then it wouldn't be so bad in this respect.  Maybe in
the future....

Unfortunately, Net::Daemon isn't really worthwhile, as it is lacking in
flexibility (no prefork at all, and threads isn't working all too well under
Perl 5.8), and NetServer::Generic is long outdated.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

Todd Vierling wrote:
> [...]
> Note that this release does not yet contain a pool-based forking 
>  dispatcher.  A couple different implementations were submitted; 
>  I plan to review them  now, take the best concepts from them, 
>  and implement an official pool  dispatcher in 0.95.

Have you considered making PMilter "Net::Server friendly" ?

IMHO it is a very good idea to make PMilter use "forking management" 
invented and maintained by somebody else in "CPAN pool".

> [...]

-- 
Andrzej [en:Andrew] Adam Filip an...@pr... an...@xl...
Home Page http://anfi.homeunix.net/ [ PageRank 6 ]

Yup, I'm still here, folks.  Sorry for the lag here, but in times like these
a blast of deadlines from The Workplace can get in the way of more
interesting things like open source projects.

Rest assured that I'm trying hard to find the time to integrate some process
pooling ideas, fix a couple outstanding bugs, and get a new release out.
I'll at least respond to open posts and mails this weekend to get my backlog
cleaned up a bit.

Sorry about that, y'all.  8-)

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

At 11:07 AM +1000 10/6/04, Rob Casey wrote:

>When this occurs, do you subsequently see issues with Sendmail being unable
>to establish a connection with the milter?  This would potentially explain
>some of the behaviour which I see on some of our front-end boxes when under
>substantial load.

I think that depends on what dispatcher you are using and what's 
triggering the EOF in stream message. I can easily see that if enough 
processes die() at once in a loaded situation that you may have 
trouble re-forking enough processes in time and sendmail may timeout 
waiting for the milter an move on - I think we've seen this happen 
twice (ever), so it doesn't seem to happen too often to our customers.

Is the messgae you're seeing sendmail log a "Connection refused" or a 
"timeout before data read"?

For example, if the SMTP peer goes away, sendmail seems to handle 
things reasonably well, and not complain about "write(Q) returned -1, 
expected 5: Broken pipe" like it often does as discussed already on 
this list:

Jun  9 11:58:42 laspinas sm-mta[7252]: i591wNmv007252: collect: 
premature EOM: u
nexpected close
Jun  9 11:58:42 laspinas sm-mta[7252]: i591wNmv007252: collect: 
unexpected close
  on connection from pomr5s2.daltron.com.pg, sender=<sam...@tr...>
Jun  9 11:58:42 laspinas sm-mta[7252]: i591wNmv007252: 
from=<samoa_nana@treasury
.gov.pg>, size=7413, class=0, nrcpts=1, proto=ESMTP, daemon=MTA, 
relay=pomr5s2.d
altron.com.pg [202.1.53.54]
Jun  9 11:58:42 laspinas isp_milter[6889]: EOF in stream
Jun  9 11:58:42 laspinas isp_milter[6889]: shutting down

Also meant to mention - in the signal handler in a number of 
dispatchers there is code like the following:

	local $SIG{$siginfo} = sub {
		print STDERR "Number of active children: $nchildren\n"
			if ($DEBUG > 0);
	};

Is there any reason this can't be a warn() instead so daemons can 
trap it with a $SIG{__WARN__} handler and syslog it? Our daemons tend 
to have code like the following so you het messages like the last two 
in the above example:

         $SIG{__WARN__} = sub {
                 my ($message) = @_;
                 chomp ($message);
                 syslog ('warning', $message);
         };
         $SIG{__DIE__} = sub {  
                 my ($message) = @_;
                 chomp ($message);
                 syslog ('crit', $message);
                 syslog ('crit', 'shutting down');
                 closelog;
         };

-- 
Andrew Rutherford      sip:an...@ia...      244 Pirie Street
Iagu Networks          tel:+61-8-8425-2255       Adelaide SA 5000
http://www.iagu.net/   mailto:an...@ia...   Australia

On 10/6/04 10:59 AM, "Andrew Rutherford" <an...@ia...> wrote:

> There's a number of comments in this thread about the message "EOF in
> stream" being a warning - but it's not, it's a die(), which makes a
> difference to pool-based dispatcher (prefork or similar, I'm
> experimenting with a dynamic pool based one) when the children keep
> dying earlier than expected.
> 
> We have some ISP customers who delay processing their RBL's until the
> "RCPT TO:" is sent, so those users who don't want RBL's don't have
> them applied. This means a lot of SMFIC_ABORT's and lots of pool
> processes die()'ing with "EOF in stream" messages.

When this occurs, do you subsequently see issues with Sendmail being unable
to establish a connection with the milter?  This would potentially explain
some of the behaviour which I see on some of our front-end boxes when under
substantial load.

Regards,
Rob

Rob Casey
Technical Manager
Bluebottle Solutions Pty Ltd
http://www.bluebottle.com/

There's a number of comments in this thread about the message "EOF in 
stream" being a warning - but it's not, it's a die(), which makes a 
difference to pool-based dispatcher (prefork or similar, I'm 
experimenting with a dynamic pool based one) when the children keep 
dying earlier than expected.

We have some ISP customers who delay processing their RBL's until the 
"RCPT TO:" is sent, so those users who don't want RBL's don't have 
them applied. This means a lot of SMFIC_ABORT's and lots of pool 
processes die()'ing with "EOF in stream" messages.

Dear Open Source developer

I am doing a research project on "Fun and Software Development" in which I kindly invite you to participate.
You will find the online survey under http://fasd.ethz.ch/qsf/. The questionnaire consists of 53 questions and you will need about 15 minutes to complete it.

With the FASD project (Fun and Software Development) we want to define the motivational significance of fun when software developers decide to engage in Open Source projects. What is special about our research project is that a similar survey is planned with software developers in commercial firms. This procedure allows the immediate comparison between the involved individuals and the conditions of production of these two development models. Thus we hope to obtain substantial new insights to the phenomenon of Open Source Development.

With many thanks for your participation,
Benno Luthiger

PS:
The results of the survey will be published under http://www.isu.unizh.ch/fuehrung/blprojects/FASD/.
We have set up the mailing list fa...@we... for this study. Please see http://fasd.ethz.ch/qsf/mailinglist_en.html for registration to this mailing list.

_______________________________________________________________________

Benno Luthiger
Swiss Federal Institute of Technology Zurich
8092 Zurich

Mail: benno.luthiger(at)id.ethz.ch
_______________________________________________________________________
On Fri, 14 May 2004, susan barnes wrote:

: All macros stay in effect from the point they are received until the end
: of the connection for the first two sets, the end of the message for the
: third (xxfi_envfrom), and just for each recipient for the final set
: (xxfi_envrcpt).

Considering PMilter was a cleanroom reimplementation, I didn't necessarily
get *all* the compatibility details right.  :)

I'll enter this into the bug tracker and plan to get it fixed before the
next Sendmail::PMilter release -- hopefully this week.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

Hello,

We are playing around with greylisting (namely Evan Harris relay-delay filter)

Standard Sendmail::Milter has shown to be unstable on our mailserver, so we are now replacing it with pmilter and found the following problem (and workaround).

The original libmilter-docs say:

smfi_getsymval returns the value of the given macro as a null-terminated string, or NULL if the macro is not defined.

All macros stay in effect from the point they are received until the end of the connection for the first two sets, the end of the message for the third (xxfi_envfrom), and just for each recipient for the final set (xxfi_envrcpt).

Pmilter however forgets macros from preceding callbacks, since it clears $this->{symbols} (line 244 of Context.pm) with each callback. This breaks compatibility with Sendmail::Milter. (The relay-delay filter needs almost all macros in its envrcpt-callback).

An easy workaround is setting:

define(`confMILTER_MACROS_ENVRCPT', confMILTER_MACROS_ENVRCPT``, $MACROS_NEEDED'')

in sendmail.cf.

If it is not possible to add this feature to PMilter, maybe it can be mentioned more clearly in the documentation.

Regards
Susan

-- 
Susan Barnes <S.B...@rr...>
(postmaster / listmaster services)
Zentrum fuer Angewandte Informatik - Universitaetsweiter Service RRZK
Universitaet zu Koeln / Cologne University        - Tel:0221-478-5594 

I thought I had best send my reply through to the list as well :-)

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

------ Forwarded Message
> From: Rob Casey <rob...@bl...>
> Date: Fri, 23 Apr 2004 10:26:13 +1000
> To: Todd Vierling <tv...@du...>
> Subject: Re: [pmilter-users] Note for milter application developers
> 
> G'Day Todd,
> 
> On 2004-04-23 10:16 AM, "Todd Vierling" <tv...@du...> wrote:
> 
>> On Fri, 23 Apr 2004, Rob Casey wrote:
>> 
>> : Where message modification actions such as addrcpt and delrcpt are
>> specified
>> : and stored in the milter context (prior to their action in the eom
>> : callback), ensure that these actions are reset prior to saving the mlfipriv
>> : context at the end of the eom callback.
>> 
>> Or, in short:  If you use setpriv() in "envfrom", "envrcpt", "header",
>> "eoh", or "body", make sure to clear all saved state in "eom".  This is
>> also true of the C API; the per-connection data sticks around all the way
>> until the connection is closed.
> 
> After working with Charles Ying's Sendmail::Milter, I had got into the habit
> of always calling setpriv() before exiting a callback.  Failure to do that
> with Charles' version of Sendmail::Milter would often result in an error along
> the lines of "Attempt to free non-existent shared string at ..."
> 
>> (Ideally, for cleanliness, it's a good idea to "setpriv(undef)" in the
>> "close" callback as well -- and do any other external-resource cleanup there
>> too.)
> 
> Yep, am doing this too.
> 
> Regards,
> Rob
> 
> 
> -- 
> Rob Casey
> Technical Manager
> Bluebottle Solutions Pty. Ltd.
> http://www.bluebottle.com/
> 

------ End of Forwarded Message

On Fri, 23 Apr 2004, Rob Casey wrote:

: Where message modification actions such as addrcpt and delrcpt are specified
: and stored in the milter context (prior to their action in the eom
: callback), ensure that these actions are reset prior to saving the mlfipriv
: context at the end of the eom callback.

Or, in short:  If you use setpriv() in "envfrom", "envrcpt", "header",
"eoh", or "body", make sure to clear all saved state in "eom".  This is
also true of the C API; the per-connection data sticks around all the way
until the connection is closed.

(Ideally, for cleanliness, it's a good idea to "setpriv(undef)" in the
"close" callback as well -- and do any other external-resource cleanup there
too.)

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

G'Day All,

Whilst this may be redundant for some developers, a problem which I have run
across in my own development code-base and have been in discussion with Todd
about privately may be of interest to others on this list.

Where message modification actions such as addrcpt and delrcpt are specified
and stored in the milter context (prior to their action in the eom
callback), ensure that these actions are reset prior to saving the mlfipriv
context at the end of the eom callback.  This is because failure to reset
these modification actions may result in actions from one message being
enacted on others sent within the same SMTP session.

The case in point - In development of code for Bluebottle, recipient
modification actions are specified in the rcpt callback, stored in the
mlfipriv context and then enacted in the eom callback.  The failure to reset
these modification actions prior to calling $ctx->setpriv at the end of the
eom callback resulted in much stress trying to determine why certain
messages were being delivered to the incorrect destination.  It turned out
that these delivery were the result of application of previously stored
recipient modification actions still stored in the mlfipriv context.

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

G'Day All,

In employing the Sendmail::Milter interface to the Sendmail::Pmilter
package, it appears that the max_interpreters and max_requests arguments to
main() are not honoured.  Upon further investigation, it appears that these
arguments are similarly ignored by Sendmail::Pmilter::main.

I have added a short snippet of code to lib/Sendmail/Pmilter.pm to honour
these arguments and tested this successfully via the Sendmail::Milter
interface.  

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

On Tue, 30 Mar 2004, Rob Casey wrote:

: I have added a preliminary implementation for the mlfi-setsender additional
: functionality into Sendmail::Pmilter as per Feature Request 925579 (Thanks
: for that Todd!)

As an aside, because this is as yet an unblessed-by-Sendmail flag bit, use
of the setsender operation requires setting an enable variable.  This is now
described in "perldoc Sendmail::PMilter" as a note underneath the
SMFIF_SETSENDER documentation.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

G'Day All,

I have added a preliminary implementation for the mlfi-setsender additional
functionality into Sendmail::Pmilter as per Feature Request 925579 (Thanks
for that Todd!) - Whilst a preliminary implementation, I expect to be
testing this fairly rigorously over the next few days and will update the
implementation if necessary.

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

G'Day All,

I have updated the Makefile.PL in the CVS distribution to prompt for the
installation of the Sendmail::Milter compatibility interface as per feature
request 923158.  It would be appreciated if early adopters could test this
to ensure it's correct behaviour (as there is much weirdness in the depths
of ExtMakeMaker from whence better men than I have run screaming in terror
:-)

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

On Mon, 29 Mar 2004, Rob Casey wrote:

: > : main release - This extension relates to the mlfi-setsender extension which
: > : I wrote (http://www.sourceforge.net/projects/mlfi-setsender/) for modifying
: > : the SMTP envelope from address.

: Not yet.  I have submitted the patch to them upon the release of 8.12.10 but
: have not received any further feedback to date.

Added as RFE 925579.

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

On 2004-03-29 11:02 AM, "Todd Vierling" <tv...@du...> wrote:

> : main release - This extension relates to the mlfi-setsender extension which
> : I wrote (http://www.sourceforge.net/projects/mlfi-setsender/) for modifying
> : the SMTP envelope from address.
> 
> Is this extension blessed by Sendmail Inc?  That is, can I rely on bit 0x40
> to be assigned to this extension, even if only in _FFR state?

Not yet.  I have submitted the patch to them upon the release of 8.12.10 but
have not received any further feedback to date.

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

On Mon, 29 Mar 2004, Rob Casey wrote:

: For the record, the problems which I'm running into with the 0.4.x branch
: relates to collisions between milter contexts under high load.  That is,
: more than a single message appears to be acted upon by changes within a
: single context.  Very weird stuff ... I'm *really* hoping that I haven't
: uncovered a race issue in libmilter :-(

I wouldn't be surprised if there are some high load issues in pmilter
0.4/0.5.  0.9x, however, is in production at a bit more than just my home
server.

: main release - This extension relates to the mlfi-setsender extension which
: I wrote (http://www.sourceforge.net/projects/mlfi-setsender/) for modifying
: the SMTP envelope from address.

Is this extension blessed by Sendmail Inc?  That is, can I rely on bit 0x40
to be assigned to this extension, even if only in _FFR state?

(I was considering adding support for the _FFR_QUARANTINE bit too.)

-- 
-- Todd Vierling <tv...@du...> <tv...@po...>

On 2004-03-29 10:33 AM, "Todd Vierling" <tv...@du...> wrote:

> On Mon, 29 Mar 2004, Rob Casey wrote:
> 
> : I've had a look through the code and see that Todd has implemented the
> : pre-forker in a slightly different manner to that which I have in our
> : equivalent Pmilter::Server module from the 0.40 release (and provided to
> : Todd for reference).  Are there any outstanding issues that anyone is aware
> : of with this dispatcher?
> 
> I have been using it since I cleaned the reference you submitted without a
> hitch (after I fixed the termination-signal issues).  That's the one I am
> using all the time now, so it's the dispatcher likely to get the best
> shakedown by the author.  8-)

Cool.  Thanks for that Todd.

For the record, the problems which I'm running into with the 0.4.x branch
relates to collisions between milter contexts under high load.  That is,
more than a single message appears to be acted upon by changes within a
single context.  Very weird stuff ... I'm *really* hoping that I haven't
uncovered a race issue in libmilter :-(

Given the incorporation of the pre-fork dispatcher into 0.91, there is only
a single additional extension which we require in the Sendmail::Milter for
our own production usage, so it's advantageous if we can move back to this
main release - This extension relates to the mlfi-setsender extension which
I wrote (http://www.sourceforge.net/projects/mlfi-setsender/) for modifying
the SMTP envelope from address.

Regards,
Rob

-- 
Rob Casey
Technical Manager
Bluebottle Solutions Pty. Ltd.
http://www.bluebottle.com/

2004	Jan	Feb (3)	Mar (23)	Apr (4)	May (3)	Jun (4)	Jul	Aug (5)	Sep	Oct	Nov	Dec
2005	Jan	Feb	Mar	Apr (1)	May (1)	Jun	Jul	Aug	Sep (1)	Oct (1)	Nov (2)	Dec (2)
2006	Jan	Feb	Mar (1)	Apr	May (1)	Jun	Jul (3)	Aug (3)	Sep (1)	Oct (4)	Nov	Dec (3)
2007	Jan (2)	Feb	Mar (2)	Apr	May	Jun	Jul	Aug (1)	Sep	Oct	Nov	Dec
2008	Jan	Feb (2)	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2009	Jan	Feb (4)	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2014	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug (1)	Sep	Oct	Nov	Dec

pmilter-users Mailing List for PMilter: Pure-Perl Milter (Page 2)

pmilter-users — Two-way discussion list for PMilter users