lurker-users — Lurker user support

Re: [Lurker-users] Lurker 2 feedback and various questions

[Julian M. <ju...@me...>]

Wesley & Jonas,

five years ago I wrote in this thread ...

  http://www.terpstra.ca/lurker/message/20060426.192614.7b0ef322.en.html :

> The Apache 2 docs have to say the following about per-directory
> RewriteRules:
> 
> | Unbelievably mod_rewrite provides URL manipulations in per-directory
> | context, i.e., within .htaccess files, although these are reached a
> | very long time after the URLs have been translated to filenames. It
> | has to be this way because .htaccess files live in the filesystem, so
> | processing has already reached this stage. In other words: According
> | to the API phases at this time it is too late for any URL
> | manipulations. To overcome this chicken and egg problem mod_rewrite
> | uses a trick: When you manipulate a URL/filename in per-directory
> | context mod_rewrite first rewrites the filename back to its
> | corresponding URL (which is usually impossible, but see the
> | RewriteBase directive below for the trick to achieve this) and then
> | initiates a new internal sub-request with the new URL. This restarts
> | processing of the API phases.
> |
> | Again mod_rewrite tries hard to make this complicated step totally
> | transparent to the user, but you should remember here: While URL
> | manipulations in per-server context are really fast and efficient,
> | per-directory rewrites are slow and inefficient due to this chicken
> | and egg problem. But on the other hand this is the only way
> | mod_rewrite can provide (locally restricted) URL manipulations to the
> | average user.
> 
> That's why I'd like to avoid them.
> 
> I researched the issue, and in a per-server RewriteRule,
> %{REQUEST_FILENAME} oddly does NOT contain the filename as mapped by
> the "/lurker" Alias, but only the URI path, e.g. "/lurker/splash/index
> .en.html".  It seems that the Alias is NOT processed before the
> per-server RewriteRule.
> 
> So as far as I can see, the following is the only way to avoid a per-
> directory RewriteRule.  Note that I have split the /var/www/lurker
> frontend cache into several caches into /var/lib/lurker/www/<frontend>.
> 
>   ScriptAliasMatch ^/lurker/((lurker|jump|keyword|bounce)\.cgi)$ /usr/lib/cgi-bin/lurker/$1
>   Alias /lurker /var/lib/lurker/www/<frontend>
>   RewriteCond /var/lib/lurker/www/<frontend>/$1 !-s
>   RewriteRule \
>     ^/lurker/((attach|list|mbox|message|mindex|search|splash|thread|zap)/[^/]+)$ \
>     /lurker/lurker.cgi \
>     [L,PT,E=LURKER_CONFIG:/etc/lurker/lurker.conf,E=LURKER_FRONTEND:/var/lib/lurker/www/<frontend>/$1]
> 
> Does anyone know a nicer way to avoid a per-directory RewriteRule?

I have now figured out how to avoid having to specify the frontend path
more than once and also avoid a per-directory RewriteRule (and thus the
performance penalty):

  ScriptAlias /lurker-cgi/ /usr/lib/cgi-bin/lurker/

  RewriteRule ^/lurker/(.*)$ - \
    [E=LURKER_CONFIG:/etc/lurker/lurker.conf,E=LURKER_FRONTEND:/var/lib/lurker/www/<frontend>/$1]
  RewriteCond %{ENV:LURKER_FRONTEND} !-s
  RewriteRule ^/lurker/(attach|list|mbox|message|mindex|search|splash|thread|zap)/[^/]+$ /lurker-cgi/lurker.cgi [L,PT]
  RewriteRule ^/lurker/ %{ENV:LURKER_FRONTEND} [L]

Feel free to add that to Lurker's docs and/or apache.conf.

-Julian

Re: [Lurker-users] Lurker web interface broken after upgrade

[Jenny H. <hop...@gm...>]

On 22 June 2011 12:53, Wesley W. Terpstra <we...@te...> wrote:
> On Wed, Jun 22, 2011 at 11:39 AM, Jenny Hopkins <hop...@gm...>
> wrote:
>>
>> I've just upgraded a server running lurker from lenny to squeeze.  The
>> web interface to my mailing list archives is now broken:
>>
>> Log file reports: "File does not exist: /var/www/lurker/imgs, referer:
>
> Well, a quick look in the debian lurker package shows that the images are in
> /use/share/lurker/www/imgs. Also, there's a symlink from ui to
> /etc/lurker/ui which includes the XSL templates to generate HTML.
> Presumably you refer to the wrong path (/var/www/lurker/imgs) in your apache
> configuration.
> I guess the debian package moved these files to conform to the LHS and
> updated this /etc/lurker/apache.conf with the new path, but you copied the
> config (or wrote your own) which then didn't get updated on upgrade.
>>
>> some of the links lead to a blank page.
>
> ... a symptom of the missing XSL files.
> My suggestion: include /etc/lurker/apache.conf in your server configuration.
>
>

Wesley,

You were correct.  The lurker.conf.local also needed the new path updated.
All seems well now. :-)

Thanks,

Jenny

[Lurker-users] Lurker web interface broken after upgrade

[Wesley W. T. <we...@te...>]

On Wed, Jun 22, 2011 at 11:39 AM, Jenny Hopkins <hop...@gm...>wrote:

> I've just upgraded a server running lurker from lenny to squeeze.  The
> web interface to my mailing list archives is now broken:

Log file reports: "File does not exist: /var/www/lurker/imgs, referer:
>

Well, a quick look in the debian lurker package shows that the images are in
/use/share/lurker/www/imgs. Also, there's a symlink from ui to
/etc/lurker/ui which includes the XSL templates to generate HTML.

Presumably you refer to the wrong path (/var/www/lurker/imgs) in your apache
configuration.
I guess the debian package moved these files to conform to the LHS and
updated this /etc/lurker/apache.conf with the new path, but you copied the
config (or wrote your own) which then didn't get updated on upgrade.

some of the links lead to a blank page.


... a symptom of the missing XSL files.

My suggestion: include /etc/lurker/apache.conf in your server configuration.

[Lurker-users] Lurker web interface broken after upgrade

[Jenny H. <hop...@gm...>]

Hullo there,

I've just upgraded a server running lurker from lenny to squeeze.  The
web interface to my mailing list archives is now broken:
http://www.aleph1.co.uk/lurker/list/yaffs.html

Log file reports: "File does not exist: /var/www/lurker/imgs, referer:
http://www.aleph1.co.uk/lurker/message/20110621.023517.1b23293a.en.html"
type messages, but as well as lack of images some of the links lead to
a blank page. In addition, the links from the front page then cease
being hyperlinks and appear to turn into ordinary text.

I'd be very grateful if anyone on list can prod me in the right
direction for troubleshooting.

Thanks,
Jenny

Re: [Lurker-users] Back up of the Lurker system

[Wesley W. T. <we...@te...>]

On Fri, Mar 11, 2011 at 10:30 PM, Robert Woodworth
<ro...@a1...>wrote:

> This is embarrassing, I really SHOULD know how to do this, but I don’t.
>
> How do I lock this file ?
>

>From a shell script:
flock -e /var/lib/lurker/db.writer -c your-script-that-runs-tar-or-whatever


>  When you say block level, you mean something like “dump” or “tar” ?
>

I meant something like LVM.

Re: [Lurker-users] Back up of the Lurker system

[Wesley W. T. <we...@te...>]

On Fri, Mar 11, 2011 at 9:53 PM, Robert Woodworth
<ro...@a1...>wrote:

> Whats the best way to do backups of the Lurker system?
>
>
>
> If I run a cron job to copy the database over to another location, how do I
> ensure that my backup and an indexing process do not collide ?
>

If you lock the file /var/lib/lurker/db.writer during your backup of the
lurker database, this will stop any new messages from being indexed.

Another option is a filesystem or block-level snapshot. At any given instant
the lurker database is consistent, so any sort of snapshotting technology
will work.

Re: [Lurker-users] Back up of the Lurker system

[Robert W. <ro...@a1...>]

This is embarrassing, I really SHOULD know how to do this, but I don't.

 

How do I lock this file ?

 

When you say block level, you mean something like "dump" or "tar" ?

 

 

From: Wesley W. Terpstra [mailto:we...@te...] 
Sent: Friday, March 11, 2011 1:25 PM
To: Robert Woodworth
Cc: lurker-users
Subject: Re: [Lurker-users] Back up of the Lurker system

 

On Fri, Mar 11, 2011 at 9:53 PM, Robert Woodworth <ro...@a1...>
wrote:

Whats the best way to do backups of the Lurker system?

 

If I run a cron job to copy the database over to another location, how do I
ensure that my backup and an indexing process do not collide ?

 

If you lock the file /var/lib/lurker/db.writer during your backup of the
lurker database, this will stop any new messages from being indexed.

 

Another option is a filesystem or block-level snapshot. At any given instant
the lurker database is consistent, so any sort of snapshotting technology
will work.

 

[Lurker-users] Back up of the Lurker system

[Robert W. <ro...@a1...>]

Whats the best way to do backups of the Lurker system?

 

If I run a cron job to copy the database over to another location, how do I
ensure that my backup and an indexing process do not collide ?

Re: [Lurker-users] lurker db tunables

[Wesley W. T. <we...@te...>]

On Sun, Jan 9, 2011 at 1:26 AM, Shane W <shane-from-lists.sourceforge.net@
csy.ca> wrote:

> First, the database is often creating new files and
> deleting which isn't that efficient for rsync to the
> hosting provider. How does lurker decide when to roll a new
> file vs append to an existing when adding a message to an
> archive.
>

The database files in lurker are immutable. Every time lurker-index is run,
a new database file is 'rolled up' with other existing files. Given the
nature of the index lurker uses, this is quite efficient for magnet disks.

As for rsync bandwidth, I wouldn't worry too much. The new database files
created correspond to new email. On average, a given key in the database
will move through log(N) otherwise unchanging database files, which means it
will be rsync'd quite a bit less often than it would be in say, a BTree or
hash table where nearby changes to the file cause rsync to recopy it.

If I were you, my main concern about rsync would be that it doesn't take a
consistent snapshot. If lurker-index runs while rsync runs, you can get a
database summary file that lists files which rsync didn't copy. When I take
backups, I use LVM to snapshot the disk and backup from that. Another
possibility would be to lock the file 'db.writer' while rsync runs, which
will block lurker-index processes which must acquire this lock.


> The other issue is database size. I currently have a number
> of lists being archived and the database is around 8%
> larger than the compressed mbox size. Is that typical?
>

Sounds about right. The database includes most of the content of the
original email messages in addition to various meta-data.


> I ask about tunables as $LIBDIR/db looks like:
> 1 8192 255
>

First # is the database format version number (1).
Second # is the block size (8192).
Third number is the maximum key length (255).

I wouldn't recommend changing the key length. I suppose you could change the
block size, but be advised that values != 8192 haven't really been tested.

I would be surprised if lurker requires much resources on a modern system,
such that you need to tune it. It was built to run debian-scale mailing
lists on computers which are now 10 years old... Do you have a workload
where there is a performance problem?

I am aware of several changes that would greatly increase the database
performance, but never implemented them as they have never been necessary.

Re: [Lurker-users] Mail list limits

[Robert W. <ro...@a1...>]

Boy have I made a mess of this thing.

I wish some of the details that Im getting in this mail exchange were in the
manpage.

 

1st of all, one of those l's was meant to be a i

 

OK so those are mboxes rather than databases in /var/lib/lurker

This is a relief.

 

Heres the problem:

I modified procmailrc to send directly to lurker -index.

I also ran lurker-index on the old mailbox files I used to route procmail
to.

1st lists no problem.

Support list, the lurker-index process hangs.

I also think I may have indexed 1 of the lists more than once, which you
cautioned me not to do.

 

If a list is indexed too many times, will lurker-regenerate fix it ?

(You said that overindexing leads to oversized databases)

 

Since I have the old mbox, and I have the zipped one that contains a mix of
the old and new,

Is there a way to merge and de duplicate and then index the result ?

 

 

 

 

 

From: Wesley W. Terpstra [mailto:we...@te...] 
Sent: Saturday, January 08, 2011 5:51 AM
To: Robert Woodworth
Cc: lurker-users
Subject: Re: [Lurker-users] Mail list limits

 

On Fri, Jan 7, 2011 at 11:28 PM, Robert Woodworth <ro...@a1...>
wrote:

Now that everything goes directly to lurker-index, if something goes awry
with the database, Im screwed right ?

 

The files /var/lib/lurker/your-mailing-list are gzip'd mboxes. Do not modify
them or open them in any sort of mail client or you will mess things up.
However, zcat list > a-copy-of-the-list will get you a valid mailbox.

 

So when I ran " lurker-index -l support -I support ; lurker-index -l support
-I oldsupport", I seem to have corrupted the index.

 

I don't understand why you have two '-l' options?

 

Is there a repair for the index ?  Or am I stuck with it ?

 

You can rebuild the database from those internal mbox copies lurker keeps
using lurker-regenerate.

Re: [Lurker-users] Mail list limits

[Wesley W. T. <we...@te...>]

On Fri, Jan 7, 2011 at 11:28 PM, Robert Woodworth
<ro...@a1...>wrote:

> Now that everything goes directly to lurker-index, if something goes awry
> with the database, Im screwed right ?
>

The files /var/lib/lurker/your-mailing-list are gzip'd mboxes. Do not modify
them or open them in any sort of mail client or you will mess things up.
However, zcat list > a-copy-of-the-list will get you a valid mailbox.

So when I ran “ lurker-index –l support –I support ; lurker-index –l support
> –I oldsupport”, I seem to have corrupted the index.
>
>
I don't understand why you have two '-l' options?


> Is there a repair for the index ?  Or am I stuck with it ?
>

You can rebuild the database from those internal mbox copies lurker keeps
using lurker-regenerate.

Re: [Lurker-users] Mail list limits

[Wesley W. T. <we...@te...>]

On Wed, Dec 22, 2010 at 9:06 AM, Wesley W. Terpstra <we...@te...>
wrote:
> On Wed, Dec 22, 2010 at 2:30 AM, Robert Woodworth
> <ro...@a1...> wrote:
>> Ive been experimenting and somewhere close to 3000 messages is where it
>> starts to hang.
> How are you feeding messages to lurker-index, exactly?

Nevermind, I found your earlier post. You are running fetchmail to get mail
for multiple mailing lists, splitting it into mailboxes based on the 'To:'
field and then running lurker over the entire resulting mailbox.

I can see now why lurker is getting slower and slower.

I think from the context of your setup that you still misunderstand how
'lurker-index' is intended to be run. You are supposed to run it on messages
as they arrive, >ONE TIME ONLY<. Every time you feed a message to lurker it
puts it into the database again. That means that if you are indexing the
same mailbox over and over, your mailbox is getting very very large as each
message gets inserted repeatedly. lurker assumes that those old message were
delivered to the list again and faithfully rearchives them for you.

So, consider your example where you run fetchmail followed by lurker every
10 minutes. If there are 3000 messages in that mailbox, after one day
lurker-index will have been run 24*6=144 times. You now have a database that
is 144* the size it should be.

Here's how you should be using lurker-index:

In a typical setup with one email address per list, the MTA delivers the
incoming messages one-at-a-time as they arrive to lurker-index with the '-m'
option. Each time a mail arrives, it (and only it) gets routed to
lurker-index.

In a setup with fetchmail with one email per list, fetchmail gets the email
and feeds the (new) messages to lurker-index.

In a setup with fetchmail with one email shared between all lists (this is
your setup), fetchmail gets the email and feeds it to procmail. procmail
splits the email stream back into the different lists and feeds each email
to lurker-index.

The problem with your setup is that your procmail isn't feeding messages to
lurker-index; it is feeding them to a mailbox. Then, you are running
lurker-index on the entire mailbox instead of only the new messages.

In my earlier response to your postings, I asked you to use this:

A perhaps better solution is to feed the mail directly to lurker-index
> from your procmail rule.
> /usr/share/doc/lurker/README.procmail describes this setup.


>
:0 w
> * ^X-Mailing-List: <exa...@li...>.*
> | lurker-index -l example-list -m
>

There are two key differences here between my proposal and what you are
doing.

1) I split up the email based on a reliable header (X-Mailing-List) instead
of the 'To' field. Please, find the header the your mailing lists add and
filter on that instead.
2) Most importantly, the rule feeds messages directly to lurker-index using
the procmail pipe syntax (|). This means that each new email gets routed to
lurker-index instead of a mailbox.

Due to point #2, lurker-index only gets invoked on the new messages. There
is no need to run lurker-index after running fetchmail; lurker-index already
got run by procmail where necessary.

I hope this clears things up.

As for the database corruption, even using lurker-index incorrectly should
not be causing this. I'm guessing it has something to do with having
multiple lurker-indexes blocking at the same time and I'm seeing if I can
reproduce the problem.

[Lurker-users] Mail list limits

[Robert W. <ro...@a1...>]

 

I think I found the problem that was causing one of my lists to be replaced
with httpd errors.

Somewhere around 3000 messages, the lurker index process hangs.

The cron kicks off another and another with all of them hanging and the
database gets corrupted.

 

Ive been experimenting and somewhere close to 3000 messages is where it
starts to hang.

100 no prob. 200 no prob although it takes nearly a minute.

Even 2500 does OK but taking over a minute.

I don't mind is taking some time, but if I go off for 8 hours and its still
not done (run manually) I have to suspect something is wrong.

 

My temporary solution is to use mutt to break up the mail folder into
smaller chunks so as not to choke Lurker.

I haven't a clue how Im going to be able to automate this

(break mail folder on logical boundaries like 1st half of month etc, update
the lurker.conf file to reflect everything etc)

 

Whats the most messages in an archive that anyone here has had under Lurker
?

 

Re: [Lurker-users] A new problem...

[Wesley W. T. <we...@te...>]

On Fri, Dec 17, 2010 at 12:33 AM, Robert Woodworth
<ro...@a1...> wrote:
> Every 10 minutes fetchmail brings the mail over from the mail appliance,
> And procmail sorts it into folders based on the "To:" field.
> lurker-index is called for each of these folders and things pretty much
> work.

A perhaps better solution is to feed the mail directly to lurker-index
from your procmail rule.
/usr/share/doc/lurker/README.procmail describes this setup.

:0 w
* ^X-Mailing-List: <exa...@li...>.*
| lurker-index -l example-list -m

> Heres the new problem:
> ======
> Internal Server Error

This is an error from apache, not lurker. I suppose if the lurker.cgi
exitted with non-zero status, it might cause this, but that would show
up in the error.log.

> Its only 1 list that does this.  (177mb & growing) Is there a folder size limit on this ?

Lurker has no file size limits.

> Rebooting the machine clears everything up.
> Ive found that service httpd restart works just as well and is less
> disruptive.

Sounds like maybe your computer has bad RAM or a buggy apache.

Usually the error.log should give you more information.

[Lurker-users] A new problem...

[Robert W. <ro...@a1...>]

We were just about ready to discuss placing this thing into actual
production
when a new problem popped up.

Every 10 minutes fetchmail brings the mail over from the mail appliance,
And procmail sorts it into folders based on the "To:" field.
lurker-index is called for each of these folders and things pretty much
work.

It turns out, the order of the flags in fetchmail is critical, but we solved
that
last week (lots of hours lost in trial & error)

Heres the new problem:
======
Internal Server Error

The server encountered an internal error or misconfiguration and was unable
to complete your request.

Please contact the server administrator, root@localhost and inform them of
the time the error occurred, and anything you might have done that may have
caused the error.

More information about this error may be available in the server error log.

Apache/2.2.3 (CentOS) Server at 192.168.32.202 Port 80
======
Its only 1 list that does this.  (177mb & growing)
Is there a folder size limit on this ?

I cant find anything useful in the httpd logs other than it griping about no
favicon.
(I finally touched favicon this morning just to shut it up)

I have a monitoring script that runs a wget every hour for each of the lists
and 
I get a result of 0 normally 
(funny, if I point it at a non existent file I still get a result 0)
When the support list goes on the blink, about twice a day, the result code
for that 1 list goes to 8.

Rebooting the machine clears everything up.
Ive found that service httpd restart works just as well and is less
disruptive.

The problem is definitely in httpd, but what is it that Lurker does to chuck
shoes into the gears?

There is nothing in /var/log/lurker

Any ideas?







-----Original Message-----
From: Gerald Livingston [mailto:ger...@sy...] 
Sent: Wednesday, December 01, 2010 7:36 PM
To: lur...@li...
Subject: Re: [Lurker-users] All lists come out identical

Sean Straw has some good info on his site.

http://www.professional.org/procmail/disclaimer.html

Search for "sandbox" on the page and click the link. It's not an easy
read but is VERY informative. 

You could adapt this:

http://markmail.org/message/itmx5xyygrmopks5

and drop the date part of the folder name to save each list's email
into its own mbox for indexing. I found that on a quick search through
the markmail archive. You may want to read through the entire thread
it's related to and see where it went, I didn't.

Gerald

On Wed, 1 Dec 2010 17:51:13 -0800
"Robert Woodworth" <ro...@a1...> wrote:

> The problem is that I will eventually have over a dozen lists and my
> mail appliance has a per user license.
> No way management is going to play that.
> The lists being archived are internal so the headers wont change.
> 
> Now if only I could make procmail work.  
> Been screwing with it since the early 90s and I STILL cant make the
> piece of *&^%$#@! Work !
> 
> -----Original Message-----
> From: Wesley W. Terpstra [mailto:we...@te...] 
> Sent: Wednesday, December 01, 2010 2:30 PM
> To: Robert Woodworth
> Cc: lurker-users
> Subject: Re: [Lurker-users] All lists come out identical
> 
> On Wed, Dec 1, 2010 at 10:29 PM, Robert Woodworth
> <ro...@a1...> wrote:
> > OK so its all based on the mail box.
> > So I should be able to use procmail to sort the mailbox into list
> > boxes
> and
> > then index the list boxes separately ?
> 
> Just make separate email accounts, one per list.
> 
> It's simpler and more robust. List-ID headers change.

-- 
I am on the email list. Unless you are sending information which should
not be shared with everyone else please reply to the list, not directly
to me.

----------------------------------------------------------------------------
--
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Lurker-users mailing list
Lur...@li...
https://lists.sourceforge.net/lists/listinfo/lurker-users

Re: [Lurker-users] All lists come out identical

[Gerald L. <ger...@sy...>]

Sean Straw has some good info on his site.

http://www.professional.org/procmail/disclaimer.html

Search for "sandbox" on the page and click the link. It's not an easy
read but is VERY informative. 

You could adapt this:

http://markmail.org/message/itmx5xyygrmopks5

and drop the date part of the folder name to save each list's email
into its own mbox for indexing. I found that on a quick search through
the markmail archive. You may want to read through the entire thread
it's related to and see where it went, I didn't.

Gerald

On Wed, 1 Dec 2010 17:51:13 -0800
"Robert Woodworth" <ro...@a1...> wrote:

> The problem is that I will eventually have over a dozen lists and my
> mail appliance has a per user license.
> No way management is going to play that.
> The lists being archived are internal so the headers wont change.
> 
> Now if only I could make procmail work.  
> Been screwing with it since the early 90s and I STILL cant make the
> piece of *&^%$#@! Work !
> 
> -----Original Message-----
> From: Wesley W. Terpstra [mailto:we...@te...] 
> Sent: Wednesday, December 01, 2010 2:30 PM
> To: Robert Woodworth
> Cc: lurker-users
> Subject: Re: [Lurker-users] All lists come out identical
> 
> On Wed, Dec 1, 2010 at 10:29 PM, Robert Woodworth
> <ro...@a1...> wrote:
> > OK so its all based on the mail box.
> > So I should be able to use procmail to sort the mailbox into list
> > boxes
> and
> > then index the list boxes separately ?
> 
> Just make separate email accounts, one per list.
> 
> It's simpler and more robust. List-ID headers change.

-- 
I am on the email list. Unless you are sending information which should
not be shared with everyone else please reply to the list, not directly
to me.

Re: [Lurker-users] All lists come out identical

[Robert W. <ro...@a1...>]

The problem is that I will eventually have over a dozen lists and my mail
appliance has a per user license.
No way management is going to play that.
The lists being archived are internal so the headers wont change.

Now if only I could make procmail work.  
Been screwing with it since the early 90s and I STILL cant make the piece of
*&^%$#@! Work !

-----Original Message-----
From: Wesley W. Terpstra [mailto:we...@te...] 
Sent: Wednesday, December 01, 2010 2:30 PM
To: Robert Woodworth
Cc: lurker-users
Subject: Re: [Lurker-users] All lists come out identical

On Wed, Dec 1, 2010 at 10:29 PM, Robert Woodworth
<ro...@a1...> wrote:
> OK so its all based on the mail box.
> So I should be able to use procmail to sort the mailbox into list boxes
and
> then index the list boxes separately ?

Just make separate email accounts, one per list.

It's simpler and more robust. List-ID headers change.

Re: [Lurker-users] All lists come out identical

[Wesley W. T. <we...@te...>]

On Wed, Dec 1, 2010 at 10:29 PM, Robert Woodworth
<ro...@a1...> wrote:
> OK so its all based on the mail box.
> So I should be able to use procmail to sort the mailbox into list boxes and
> then index the list boxes separately ?

Just make separate email accounts, one per list.

It's simpler and more robust. List-ID headers change.

Re: [Lurker-users] All lists come out identical

[Robert W. <ro...@a1...>]

OK so its all based on the mail box.
So I should be able to use procmail to sort the mailbox into list boxes and
then index the list boxes separately ?

-----Original Message-----
From: Wesley W. Terpstra [mailto:we...@te...] 
Sent: Wednesday, December 01, 2010 12:54 PM
To: Robert Woodworth
Cc: lur...@li...
Subject: Re: [Lurker-users] All lists come out identical

On Wed, Dec 1, 2010 at 9:32 PM, Robert Woodworth
<ro...@a1...> wrote:
> Im not sure how lurker is parsing the mail messages and cant find anything
> in the configs to control this.

The -l option controls where the mail is indexed.

> About every 15 minutes, after the fetchmail runs, I run:
> lurker-index -l commits -i /var/mail/lurker
> lurker-index -l software -i /var/mail/lurker
> lurker-index -l support -i /var/mail/lurker

Here's your problem: you indexed all mail to every list.

You should make a different email address for each mailing list.
lurker-commits, lurker-software, lurker-support
Subscribe them to each list individually.
Then, you run lurker-index -l commits only over the mail delivered to
lurker-commits.

There is no reliable information INSIDE an email regarding its
eventual delivery. Thus, lurker indexes a message based on the account
where it arrived.

Re: [Lurker-users] All lists come out identical

[Wesley W. T. <we...@te...>]

On Wed, Dec 1, 2010 at 9:32 PM, Robert Woodworth
<ro...@a1...> wrote:
> Im not sure how lurker is parsing the mail messages and cant find anything
> in the configs to control this.

The -l option controls where the mail is indexed.

> About every 15 minutes, after the fetchmail runs, I run:
> lurker-index -l commits -i /var/mail/lurker
> lurker-index -l software -i /var/mail/lurker
> lurker-index -l support -i /var/mail/lurker

Here's your problem: you indexed all mail to every list.

You should make a different email address for each mailing list.
lurker-commits, lurker-software, lurker-support
Subscribe them to each list individually.
Then, you run lurker-index -l commits only over the mail delivered to
lurker-commits.

There is no reliable information INSIDE an email regarding its
eventual delivery. Thus, lurker indexes a message based on the account
where it arrived.

Re: [Lurker-users] All lists come out identical

[Robert W. <ro...@a1...>]

All 3 lists are identical.
Same everything.

We have a mail appliance that people get mail from.
I created a user "lurker" who is subscribed to a few mail lists for the test
install.
I run fetchmail on a cron job that brings mail over to the lurker server for
the lurker user.
When the mail arrives, I note that it all comes from the same user (the mail
list server user "cm")
The "To" field arrives as mailinglist@domain

Im not sure how lurker is parsing the mail messages and cant find anything
in the configs to control this.
I am presuming this is in the cpp code that I ran the make on.
(I never saw binary compiled cgi programs before)

Obviously lurker regards the commit messages as valid or it wouldn't load
them into the commit list.
Unfortunately, there are no support messages in the mailbox but lurker puts
the same commit messages in the support list as well.

About every 15 minutes, after the fetchmail runs, I run:
lurker-index -l commits -i /var/mail/lurker
lurker-index -l software -i /var/mail/lurker 
lurker-index -l support -i /var/mail/lurker

Is there any other info you need to figure out whats going on ?


-----Original Message-----
From: Wesley W. Terpstra [mailto:we...@te...] 
Sent: Tuesday, November 30, 2010 10:44 PM
To: Robert Woodworth
Cc: lur...@li...
Subject: Re: [Lurker-users] All lists come out identical

On Wed, Dec 1, 2010 at 5:56 AM, Robert Woodworth
<ro...@a1...> wrote:
> All went well until someone noticed that all 3 lists (different mail
lists)
> All had the SAME content in them!

That's an interesting problem I've not seen before.

Did they have only the same 'new' messages (arrived after lurker
installed) or also the same 'old' messages (imported from a mailbox
archive)?

Re: [Lurker-users] Problem while trying to use multiple frontends

[Wesley W. T. <we...@te...>]

On Wed, Dec 1, 2010 at 11:47 AM, Stefan Ritter
<ste...@cr...> wrote:
> “The requested document does not match the file lurker intends to
> generate”
>
> Here are my configs:
> <Directory /var/www/lurker>
> <Directory /var/www/lurker-list1>
> <Directory /var/www/lurker-list2>

I was sure I'd already fixed this bug in 2006! Oddly, it still exists.

Your problem is that /var/www/lurker is a prefix of
/var/www/lurker-list1. This confuses lurker.cgi into thinking the
-list1/... is the request for frontend /var/www/lurker.

You can fix this by either:
a) renaming /var/www/lurker to /var/www/lurker-all
b) building svn/HEAD lurker where I've finally committed a patch

[Lurker-users] Problem while trying to use multiple frontends

[Stefan R. <ste...@cr...>]

Hi,

I have a mailman with two lists (list1 & list2) and in addition lurker
as archiver. Everything seems to be fine, but now I need single
frontends for list1 and list2. Unfortunately I got the following error
when trying to open one of them:

“The requested document does not match the file lurker intends to
generate”

Here are my configs:

lurker's apache config
======================

ScriptAlias /cgi-lurker /usr/lib/cgi-bin/lurker
Alias /lurker /var/www/lurker

<Directory /var/www/lurker>
  Order allow,deny
  Allow from all
  
  AddType text/xml .xsl
  AddType text/xml .xml
  AddType message/rfc822 .rfc822  
  AddDefaultCharset UTF-8
  
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-s
  RewriteRule ^(attach|list|mbox|message|mindex|search|splash|thread|zap)/[^/]+$ /cgi-lurker/lurker.cgi [L,PT,E=LURKER_CONFIG:/etc/lurker/lurker.conf,E=LURKER_FRONTEND:%{REQUEST_FILENAME}]
</Directory>

Alias /lurker-list1 /var/www/lurker-list1
<Directory /var/www/lurker-list1>
  Order allow,deny
  Allow from all
  
  AddType text/xml .xsl
  AddType text/xml .xml
  AddType message/rfc822 .rfc822  
  AddDefaultCharset UTF-8
  
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-s
  RewriteRule ^(attach|list|mbox|message|mindex|search|splash|thread|zap)/[^/]+$ /cgi-lurker/lurker.cgi [L,PT,E=LURKER_CONFIG:/etc/lurker/lurker.conf,E=LURKER_FRONTEND:%{REQUEST_FILENAME}]
</Directory>

Alias /lurker-list2 /var/www/lurker-list2
<Directory /var/www/lurker-list2>
  Order allow,deny
  Allow from all
  
  AddType text/xml .xsl
  AddType text/xml .xml
  AddType message/rfc822 .rfc822  
  AddDefaultCharset UTF-8
  
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-s
  RewriteRule ^(attach|list|mbox|message|mindex|search|splash|thread|zap)/[^/]+$ /cgi-lurker/lurker.cgi [L,PT,E=LURKER_CONFIG:/etc/lurker/lurker.conf,E=LURKER_FRONTEND:%{REQUEST_FILENAME}]
</Directory>


lurker.conf.local
=================

group = Lists
 	heading = Lists

	list = list1
		title[en] = list 1
		title[de] = list 1
		language = de
		address = ***censored***
		link = https://***censored***/cgi-bin/mailman/listinfo/list1
		description = list 1

	list = list2
		title[en] = list 2
		title[de] = list 2
		language = de
		address = ***censored***
		link = https://***censored***/cgi-bin/mailman/listinfo/list2
		description = list 2

frontend = /var/www/lurker

frontend = /var/www/lurker-list1
	allow_list = list1

frontend = /var/www/lurker-list2
	allow_list = list2


Option in mm_cfg.py
===================

PRIVATE_EXTERNAL_ARCHIVER = '/usr/bin/lurker-index-lc -l %(listname)s -m'


Do someone know what I am doing wrong? Please ask if more information needed.

Best regards,
Stefan

-- 
Stefan Ritter
Tel.: +49 (0)21 61 / 46 43-124

credativ GmbH, HRB Mönchengladbach 12080
Hohenzollernstr. 133, 41061 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz

Re: [Lurker-users] All lists come out identical

[Wesley W. T. <we...@te...>]

On Wed, Dec 1, 2010 at 5:56 AM, Robert Woodworth
<ro...@a1...> wrote:
> All went well until someone noticed that all 3 lists (different mail lists)
> All had the SAME content in them!

That's an interesting problem I've not seen before.

Did they have only the same 'new' messages (arrived after lurker
installed) or also the same 'old' messages (imported from a mailbox
archive)?

[Lurker-users] All lists come out identical

[Robert W. <ro...@a1...>]

I set up to demo Lurker for the engineering dept this morning.

All went well until someone noticed that all 3 lists (different mail lists)
All had the SAME content in them!

Suffice it to say, it put a pall over the demo.

I cant figure out what I screwed up.
Its gotta be something simple that I overlooked.

An example is that messages that belong in the software commit list also end
up in the support list.
Since nothing has come over the support list, it SHOULD be (but isn't)
BLANK!

Any clues?