Noffle handling of multiple servers
Brought to you by:
bears
Menu
▾
▴
#17 Noffle handling of multiple servers
Noffle allows plural news servers to be specified in the noffle.conf file.
Noffle appears to properly sequence through each server when aquiring overview information.
However, noffle presumes a uniqueness to the news server domain name when aquiring user authentication information. This presumption is not necessarially wrong in most configurations, as often multiple news servers will be a multiple different accounts.
In my case, however, my ISP provides plural news accounts to the same server, each account given a small volume limit (1GB) but in aggregate my total volume is higher (20GB).
The result of this is that I want to use noffle to do two things:
1) make the 20 separate 1GB news accounts appear to be a "virtual" 20GB account to my news reader.
2) use noffle to perform header prefetching overnight or when I'm not looking a new news, so my news reader does not have to spend a lengthy time downloading headers when I do want to read news.
Noffle already works perfectly for #2 (at least so far in my testing for the last day).
Noffle also appears to work for #1, but in fact when the "news server" entry in the "server" lines of the noffle.conf file are all the same domain name, Noffle only utilizes the auth info for the first server entry and not for the current line it should be utilizing.
The Cfg_beginServEnum() and Cfg_nextServ() entries work properly, in that they sequence through the list of servers. But when Cfg_authInfo is called to obtain user/pass info, instead of using the current enumerated server, it searches for the server by name. This results, in my case, in noffle using only one out of 20 accounts to log in 20 times in a row to aquire news.
Unfortunately, my skills as a C programmer are not enough to submit a patch yet (although given time I think I could work something up) so for the moment my workaround has simply been to make 20 fake domain names in /etc/hosts and point noffle's server lines at the 20 fake domain names. A workaround, yes, a complete hack, definetly, but it does work at the moment.
Logged In: YES
user_id=184
Originator: NO
Well done with the workaround. It has absolutely never occurred to me that anyone would want to access the same server under different names, but you have a valid requirement. I assume you must be using getgroups to divide traffic between the different accounts?
Unfortunately I don't think it is always the case the Cfg_authInfo is called for the currently enumerated server, retrieving an article during online mode being the case in point. Noffle has referring to servers by name as an assumption in lots of places.
How would it be if Noffle could do your /etc/hosts trick itself? It would be simple to make the server name just a name, with the hostname to connect to a separate item. So in /etc/noffle.conf you'd have something like:
server gmane1 news.gmane.org
server gmane2 news.gmane.org
and then refer to the server as 'gmane1' or 'gmane2'. I think this could be done fairly safely in Noffle.
Logged In: YES
user_id=506858
Originator: YES
> Well done with the workaround. It has absolutely never occurred to me that
> anyone would want to access the same server under different names, but you
> have a valid requirement.
Not surprising. 99.999% of the "multiple server" requests are for two or three different servers to mix them into one combined "big-server"
I did discover one small "flaw" in my workaround. Noffle now thinks the servers are different (afterall, the "fake" domain names are different) and so using my workaround it wants to download the same headers 20 times, tossing away 19 out of the 20 because it already has the header from the first pass. So it's not a perfect solution yet.
> I assume you must be using getgroups to divide
> traffic between the different accounts?
Actually, no, I think a bit more "explanation" for what I'm trying to do is in order, as what I really want to do is not the same task that Noffle has been designed for, it just happens that "online" mode is "almost" what I'm looking for.
I've got two irritations at the moment with my usenet news. The first is that my "free" access (limited) through my ISP of 20G/month is divided into 20 1GB pieces (the 20 accounts to the same server). My solution to this irritant at the moment is to install 20 "servers" in my news reader (Pan at the moment). This works, mostly, but it's a pain overall, and it does nothing to alleive my second irritant.
The second is that I do look into some of the binaries groups. The problem with the binaries groups is the total flood of postings (seemingly 20000+ articles/hour in some). This causes Pan to take a huge amount of time downloading headers, and as Pan does not have a "schedule header download at 1am) feature, it's also a manual process. So the second thing I'd like to be able to do is to prefetch the group headers overnight when I'm not using the computer/dsl line and have them all ready and waiting for me when I do open up Pan and access things. While still having the articles I want to snatch downloaded from my ISP's server when I tell Pan to access them.
Noffle's overview + online modes provide exactly this feature. Except that in the high traffic groups, Noffle takes quite some time to store away 10-20k+ headers when it does a "-fetch". Enough so that it seems I need to fetch about every 20-30 minutes lest Noffle take enough time storing a batch of headers that the connection to the news server times out.
I realize that for this last point, in the high traffic groups, I'm using Noffle in a way it was not designed to be used. It was never designed for handling groups with a firehose of 20k+ articles per hour arriving.
The 20 minute cycle works, in that the header load is low enough that Noffle seems to be able to keep up. But as the gdb file grows in size, I'm wondering if I'll just end up hitting another performance knee where things get too slow.
Ideally, what I'd really like to have is essentially a transparent nntp proxy that round robins across my 20 1GB accounts and prefetches and caches the xover data from the ISP's server and serves that cached xover data to my news reader while passing all article requests upstream to my ISP's server. Noffle is very close to that ideal, save for the performance issues with lots of headers.
> How would it be if Noffle could do your /etc/hosts trick itself? It would
> be simple to make the server name just a name, with the hostname to connect
> to a separate item. So in /etc/noffle.conf you'd have something like:
>
> server gmane1 news.gmane.org
> server gmane2 news.gmane.org
> and then refer to the server as 'gmane1' or 'gmane2'.
That would seem to provide a similar result to my plural fake hostnames in /etc/hosts. My one question is, would Noffle see gmane1 gmane2 as different servers and try to download fresh headers from each, or would it see all the "news.gmain.org" servers as one, and only download the headers once?
Also, I forgot to mention in my first report that I'm running the 1.2.0rc1 version of Noffle.
Logged In: YES
user_id=184
Originator: NO
Sorry for the delay getting back to you, and thanks for the extra detail. I'm away from home at the moment, and currently reliant on what must be one of the flakiest wireless networks in Italy. I couldn't get it work at all yesterday, and you don't want to know how painful it is today.
I don't think I understand how you allocate accesses between different auth credentials. But I have a suggestion that I could implement quite easily - how about allow you to specify multiple user/pass combinations for a single server, and have Noffle choose one randomly whenever it needs one? You wouldn't have fine control over which to use - unless I also add a command line flag allowing specific selection - but it would do pretty much what you want, and have the benefit of Noffle knowing it is dealing only with one server.