Menu
▾
▴
#9 Does not find "old" segment files
starting with netw362.139 (new compile extension will be .160), makenl complains it will not find any segment file, despite the fact a valid segment file is still available in the master directory
renaming netw362.139 to netw362.146 did the trick.
checking for only for 3 weeks back (160, 153, 146) is an unacceptable behavior
editnl checks for files of a 7 weeks range
This is the design behavior. It can easily be changed if there are sufficient requests for it.
More reports regarding this issue have been made (in a discussion of the issue in the MAKENL_NG echo), especially by RCs where the issue is causing problems with their current nodelist operations because valid nodelist segments that are older than three weeks are not being included in new regional nodelists being generated for sending to their ZC.
Note that in one specific instance, what was present was a valid nodelist segment with the extension ".323". But the extensions that were looked for (as per the debug log) was only the following: ext='025', ext='018', & ext='011'.
Issue appears to be that the maximum number of files to be checked for is currently hard coded and is only set to "3". This is done using the constant SEARCH_MASTER in lsttool.h.
Why don't we bring it up to 7 for 3.3.4; we'll have to test it thoroughly
to ensure that it doesn't cause other issues.
I will work on it this week.
Andrew
On Tue, Jan 22, 2013 at 8:15 AM, Robert James Clay jame@users.sf.netwrote:
Related
Bugs:
#9On Tue, Jan 22, 2013 at 7:04 PM, Andrew Leary ajleary@users.sf.net wrote:
That should work.
and being tested awhile, do a release as 3.4.0. (Of version of 3.3.x
that we're at, at the time...)
Great!
--
Robert J. Clay
rjclay@gmail.com
The change has been made; but in my initial testing it is causing problems
with the wrap around to the previous year. I suspect I'm going to have to
rewrite the code that generates the list of previous extensions, possibly
by manipulating the time variable used to calculate the next extension to
be used. The 16-bit OS/2 version is currently aborting with "Unable to
create 356\1.msg" so it appears that something is clobbering the netmail
path when the oldweeks calculation extends beyond the year boundary.
This is going to take a little longer than I initially thought...
Andrew
On Thu, Jan 24, 2013 at 9:00 AM, Robert James Clay jame@users.sf.netwrote:
Related
Bugs:
#9Need an status update from Andrew Leary on this, but I'm fairly sure this has all been fixed recently. Using Uli Schroeter's CTL files seems to work as intended, for me.
This should be all set to go back 7 weeks; but I do need to test to make sure it doesn't have issues when the 7 week period wraps into the previous year.
mhh ... current segments check is a bit odd ...
only to search for 3, probably now with 7 segments starting from the current fridays (or publish day varied on another days) offset isn't that is required in practice.
As higher the checklevel is (NC -> RC -> ZC) the variations of segment files to handle becomes weaker and weaker ... segments with out-of-sync offset numbers, one downlink sends a fixed filename segment, the next one sends a weekly .067 named segment until the next change happens in the segment. A solution in these special cases is to add .* so whatever filename extension is used by the downlink, the segment gets imported. But still then, the main work, to cleanup the master directory doesn't work.
Why not move forward with an unlimited scan of all configured directories where segments can reside (upload, inbound, update, masterdir), analyse the segments datetime, analyse the admins line segments proposed publishing date / daynumber, and search for the newest segment as input ?
use this segment in the automatic numbering schema for the working directory -
if daynumber doesn't match to the upcoming publising daynumber, rename the found segment to the current daynumber
With such an extended routine, another bug/feature request, to check segments in inbound
eg CRC check can be easily included. If a crc check in the inbound directory fails, the scan for another segment can continue and probably can found the last weeks segment in the master directory
Dont count only segment files with fixed daynumbers that probably fails, instead collect all segment files, order by date, and use the newest one ...
This is possible for a future version, but it will not be done for 3.4.0. 3.4.0 (when released) should go back 7 weeks searching for segment files in the master directory. Any received segment in the UPLoads or MAIlfiles directory will be imported with the current week's extension, and thus will be found for the next 7 weeks.
ie: The extension for next Friday is .263. Net 320 is in your control file with a generic filename:
FILES
Net 320 net320
MakeNL will search UPLoads (if defined) and MAIlfiles for net320.* Any files found will be moved to net320.263 in the UPDates directory if running in test mode, or to net320.263 in the MASter directory if running in process mode. If no files are found in either UPLoads or MAIlfiles, then MakeNL will search the MASter directory for net320.263, net320.256, net320.249, net320.242, net320.235, net320.228, and net320.221. The first one found will be used.
Unfortunately, we can't depend on file date/timestamps being correct, since they aren't always preserved when the files are transferred between systems. ie: old protocols like Xmodem that don't preserve them, or BBS systems that touch the file date/time on upload.
I would like to implement CRC checking of received segments, but again, this will not be done for 3.4.0.
Regards,
Andrew
date/timestamp is not the primary resource ... its only one of several infos to gather
segments dir admin line day# filename ext day# file date/time
1 seg1 upload 263 263 2013-09-16 ...
2 seg1 upload 263 264 2013-09-17 ...
3 seg1 mail 263 263 2013-09-17 ...
4 seg1 mail 263 264 2013-09-18 ...
5 seg1 master 256 256 2013-09-13 ...
which one would you select, reviewing the table values ? 1,2,3,4 or 5 ?-)
4 is the correct one !
this is real life what happens on a masters inbound at higher levels ...
despite the fact, the mailer renamed the extension +1, there exist 4 potential
segments for the next compile, only one was the last sent in ... and this was #4
use examples of segment files w/o admin lines, use examples of segments with daily
lower level distribution where the extension gets numbered .256, .257, .258, .259, .260
ok, one more sample =:)
segments dir admin line day# filename ext day# file date/time
1 seg1 upload 263 263 2013-09-16 ...
2 seg1 upload 263 264 2013-09-17 ...
3 seg1 mail 263 263 2013-09-17 ...
4 seg1 mail 263 264 2013-09-18 ...
5 seg1 master 258 258 2012-09-14 ...
6 seg1 master 256 256 2013-09-13 ...
this sample shows, that only extensions check ends in bad results :-P
seg1 in master should no longer be there, should be removed in the /clean process
and checking only extensions by strict offset - (x * 7) numbering schema is unappropiate
if one *C wants to strictly limit by this "offset - (x * 7)" format this can be added with a switch in the config. But default should and has to be, that all segments to apply in a check. In my Z2PNT distribution 7 out of 9 segments are current. 2 segments
lacks an update since GT 6 months .... At least from one region I know, the sysop tries to recover his system ... and the current state is frozen by end of last years status
and the segment still remains at day# 335
this shouldn't be a show stopper for 3.4.0 but ... to be considered carefuly before trying to fix current daynumbering schema implementation ... ,-)
so its worth while, to think about check routines for a year switch in daynumber counting, if a plan moves forward, to replace current counting schema ...
my2c
I have tested and confirmed that the program does properly go back 7 weeks now, including handling cases where the 7 weeks wraps into the previous year. This bug is being closed as fixed, but I will put in a feature request for Uli's request to look at analyzing segment files by date.