camram-development — list for camram development

Re: [Camram-development] [hashcash] Re: blacklist to Brown list conversion

[Eric S. J. <es...@ha...>]

Eric S. Johansson, who definitely talks too much, wrote:

> to bring it back on topic, a blacklist repair service based on proof of 
> work systems could be quite useful.  Especially if we can make it 
> something that doesn't have to change the message recipient.  For 
> example, make it a transaction between blacklisted site and blacklist 
> database.   Something like "I have been blacklisted from..., I want to 
> send one message, how much?"  and after they received the price, the 
> sender can either generate the stamp or throw away the message.  After 
> some threshold or number of times of generating stamps, the stamp price 
> declines as long as there are no hits reinforcing the pricing level.  Or 
> something like that.  Yes, I am a fan boy for dynamic pricing systems 
> because that lets us get around Moore's law inflation and reward people 
> for doing good things.

thinking some more on this, I do believe the key to dealing with 
blacklists is a Brown list CBL type system driven by tokens from blocked 
sites.

on the sender side, there should be some form of proxy intercepting 
traffic between the sending MTA and the receiving MTA.  The proxy should 
be on the sending MTA side of the transaction.  If the proxy detects a 
blacklist block, it takes the message that had been sent and stuffs it 
in a queue.  The queue consumer queries the CBrL for pricing, generates 
the appropriate size token, and then delivers the message.  The CBrL 
then gives one "not blocked" response for a query about the token 
generating address.

On the other side of the equation is the number of reports of bad 
behavior.  As the number reports climb, the cost of postage goes up.  As 
the number of reports drop and the number of tokens generated go up, the 
cost of postage goes down.

Pricing for stamps in this context should be reasonably large.  I think 
that something like a 16 minute stamp would not be unrealistic in the 
very beginning.  It's large enough that it will dissuade spammers from 
sending a single message but it's small enough to make it possible for 
someone to get a "hey, something's broken" message.  And, as the pricing 
model describes, it will get better with time if you clean up your act.

I think the "easiest" way to implement this would be to take advantage 
of modern firewall rules to intercept traffic and redirect it to a proxy 
program.  The proxy front-end should be something written in something 
like C/C++ so that it will run reasonably fast for the 99% case.  The 
backend can be cobbled together out of twopenny blue pieces because, 
quite frankly, you don't care about performance when you are generating 
eight or 16 minute stamps.

The threat model is the same as other proof of work systems, i.e. 
overwhelming force applied to the problem.  Using the zombie calculator 
13 billion spam baseline and 16 minutes stamps, you'd need 441 million 
zombies.  with proper feedback, the number of zombies needed would 
increase significantly as individual zombies are identified and priced 
out of the market.  The other advantage is that stampers can't send 
anywhere near as many spam messages.

A couple of challenges involve scaling.  How does one scale such a 
service.  How does one communicate pricing information across a network 
in such a way that it can't be double spent?

I don't think the problems should dissuade one from trying because this 
is another good example of how proof of work stamps can be used in a 
positive fashion to repair a broken system.

anyway, that's my thoughts on the subject.

---eric

-- 
Speech-recognition in use.  It makes mistakes, I correct some.

[Camram-development] 2 penny blue "my aunt is a zombie from Hell" beta version 1.10

[Eric S. J. <es...@ha...>]

http://2pennyblue.org/
Important features:

Simplified install (finally using distutils)
instructions for installation.
Recipient address verification (inbound) really working.
verped address handling
more precise control of when to generate stamps (no DNS yet)
improved queuing of stamping and training operations
improved union configuration file code
extracted multiple modules from twopenny blue as general Python tools

These improvements lead to a much faster training of CRM 114, lower 
workload on the user, and just generally more reliable.

Let me know if you find any problems and if there is nothing 
significant, I'll make a wider announcement early next week.

Any questions can go either to the mailing list or to me directly.

---eric

[Camram-development] brief update

[Eric S. J. <es...@ha...>]

camram has been renamed twopenny blue after the stamp following Penny 
Black.  Make whatever inferences you wish about comparisons to the 
Microsoft Project.

In a nutshell, I've made a few major bug fixes, improved reliability of 
a few components, extracted useful Python modules to stand alone, and 
made it easier, much much easier to install.

One major improvement has been a more reliable implementation of 
recipient address verification.  In other words when e-mail enters the 
system, the twopenny blue filter queries the destination mail server (as 
specified by the postfix transport file) and determines if the address 
has a valid recipient.  If not, the message is rejected cash.  With 
postfix this seems to work okay for for everything virtual addresses, 
aliases, as well as host accounts.

The beauty of this technique is that it dumps on the floor huge amounts 
of spam without any need to filter.

Anyway, I'm coming up to a beta test in the next 24-48 hours and I just 
wanted to give folks a heads-up.

still need to get the twopenny blue website running.

---eric

[Camram-development] October 2006 update

[Eric S. J. <es...@ha...>]

since the last release of camram, there have been a few interesting 
changes.  I have improved the

speed of queue processing,

improved detection time for new entries,

reorganized the code base,

extracted non-camram dependent modules into my "mini modules" set of 
Python modules,

Attempted to improve reliability and information gathering
init.d startup/shutdown scripts,

converted install to python distutils,

improved postfix configuration so that only one IP address is needed on 
a camram enabled server.

Postfix can now use either IP address or SMTPAUTH based authentication 
with camram

e-mail address verification for inbound mail with routing based on 
postfix transport file.

coming up real soon now,

improved installation instructions

e-mail notification if address is not signed up. (I.e. sign up now for 
your free camram account, CD will be delivered to your home any day now)

DNS-based propagation of stamp value required.[1]


an interesting side note about the DNS stamp value is that it solves one 
very important problem which is the cost of getting started.  In the 
beginning, with a camram style system, when you start sending e-mail the 
friends white list is being built.  Unfortunately, this also means that 
lots of stamps that aren't really needed are generated.  By using the 
DNS method, you can eliminate unneeded stamps until there is someone to 
listen to it.

The benefit to early adopters is all the other camram filter and false 
positive elimination techniques.  As the population of stamp users 
increases (DNS advertisements), then the benefit of stamps as an 
introducer starts to have a significant effect.

Anyway, that's how I see it.

I can use some help (like always):

Guinea pig.  I need someone willing to assist incident early adopter 
testing.  Help me figure out what needs to be fixed in the installation 
instructions etc.  What has broken in the code?

Looking for a Perl guru to modify an rdd based display of e-mail server 
activity.

---eric

[1] the DNS-based verification is something I've talked about for a 
while.  I figured out how to make a variable and still be backwards 
compatible.  The basic DNS record would contain a baseline stamp value 
for the organization.  but a simple extension to the DNS record would 
add a URL which he camram system could use to determine the appropriate 
stamp sized required for an e-mail address.  I think I'm only going to 
implement the first version which is stamp in DNS record but leave room 
for the rest.  Unfortunately, I don't have time to dive into the rest of it.

[Camram-development] a brief update

[Eric S. J. <es...@ha...>]

might as well start making use of the list....


Since the last release, there have been quite a number of changes.  I
frankly I not remember all of them and when I make the next
announcement, it will just list everything that I have built into
camram.  The most recent change however was brought about because of one
user whose version of CRM 114 would lose its mind on a semi regular
basis.  He would end up with literally thousands of messages in the
spamtrap and it was just taking forever to get through all of them.  So
originally I built a Ajax a system where the browser became the queue
manager for batch submitting messages for training.  60 hours effort
over a couple of weeks was tossed out the window when I discovered it
was not reliable nor portable.  Hint: never try to use the
XMLHttpRequest call asynchronously if you are trying to relaunch the
next request from the callback handler.

The end solution was to put in yet another queue and just submit all the
request to the queue and let the system sort them out and background.
It works very well, it does search for green requests first allowing
them to be delivered quickly and processes all of the red requests in
background.

The last bug (I just discovered) is that the spamtrap limits are not
closing improperly.  For some reason they seem to be frozen.

However in the process I discovered an unpleasant "feature".  Up till
now, most implementations of camram were on the same machine as the mail
server and local mailboxes.  I'm now experimenting with pulling the
services onto a separate box and in fact even splitting camram across
multiple boxes so you can put the front-end filter (Brown list, valid
e-mail address check) on a front-end machine and put everything else on
the backend.  But the problem is how do you inform the filter that you
have a false negative?  What had been happening was I was putting the
false negative in a mailbox and then shoving a mailbox contents into
training the content filter.  Doesn't work so well when you have
different machines for mailboxes and filtering.  Ideally I should have a
  basic plug-in which would let me send the message to the camram system
for retraining.  Unfortunately, this is not looking easy.  All I want to
support his Thunderbird and Outlook.

The basic techniques for doing this include: forwarding the message,
plug-in copying the message, and drag-and-drop.  All of them suck in
different ways.

Any ideas?

---eric