#21 New Search Segments Routine

[Uli Schroeter]

JK> After speaking voice with R19C yesterday, I was informed there is
JK> still a problem with the OS/2 version of MakeNl in processing
JK> segments. As I understand it right now with v.3.4.1, it  look doesn't
JK> look for the most current Julian date for a particular file name.  As
JK> I understand the problem, it doesn't process segments older than 6
JK> weeks.

yes, this is a known issue, where i've invented ...

the search for segments routine is a bit odd ...
it still searches the todays daynumber + the 6 daynumbers of the last 6 weeks
back
if there is a daynumber (361 last valid), next back in sequence is 361-7 = 354,
say the file has the daynumber 355, makenl will not find it ...

there is one exception:
if the segment file in the control file is defined as   file.*
default definition for numbered segment files is  file. or  file
then the strict daynumbers calculation is used

the search for segment files probably requires a complete rewrite,
die allow whatever segments in the working directory, with a more analysing
routine ... search for files by mask   eg  segment.*
analyse by date, daynumber, admin line info, sort the result and pick the
latest segment

A workaround in starting with the makenl 3.4.1 is, to bring all segment files
in sync with current daynumbers schema.
so this is a one-time job, search the latest segment of each segment, rename it
once to .361 and it will be handled subsequently by makenl in each new process
week by week  =;)

eg this weeks working dir:
    segment1.361
    segment2.004
    segment3.361
    resultseg.361

after processing makenl updates the two .361 files to  segment.004 and
segment3.004 so your working dir looks like
    segment1.004
    segment2.004
    segment3.004
    resultseg.004

makenl holds a copy from the previous week and with cleanup in the control file
defined, removes the "older" segments .354 and before

If you want to use daily nodelist creation, you have to switch to segment.*
definition, with whatever daynumbering ...
but then the cleanup no longer works
and you have to clean the working directory manualy   :-P

related bugs (closed)

https://sourceforge.net/p/makenl/bugs/9/

and discussion about "search for old segments" starts here:

https://sourceforge.net/p/makenl/bugs/9/?page=1

as the ticket has been closed, the request has been lost in development :-P

The https://sourceforge.net/p/makenl/bugs/9/ fix (editnl checks for files of a 7 weeks range instead of 3) can be seen as a workaround, but not as the final solution.

Previous discussion started:
https://sourceforge.net/p/makenl/bugs/9/?limit=10&page=1#5d67
https://sourceforge.net/p/makenl/bugs/9/?limit=10&page=1#739a
https://sourceforge.net/p/makenl/bugs/9/?limit=10&page=1#718d

so reopened as new bug/feature request following https://sourceforge.net/p/makenl/bugs/9/?limit=10&page=1#0c60

[Uli Schroeter]

some more input ....

 WD> As you know, the numbering poses no problem here (as in 80+% of the
 WD> nodelist) so "If it isn't broke, don't fix it".

fine if it works for you  =;)

what I have in mind is similar to the ALLOWunpub switch ...

having 2 routines available ...

a. if no switch is set, use old routines, and live with the flooded
   segmentfiles working dir or living with the missed segments in
   strict filename searches
b. a new routine, that is capable to search for a given filemask
   of one segment (for each segment) no matter of fileextension
    1. analyse file date
    2. analyse file daynumber extension
    3. analyse admin line (if available)
   If multiple valid segments are found, bring them in a chronicle order,
   with cleanup, leave 1 or 2 'current' segments, update the latest segment
   to current daynumber extension

   the advantage: every daynumber segment counts, no more missing segments
   caused by wrong daynumbers, no more yearly wrapping problems, as the
   analysed segments have to be sorted in an internal table by datetime
   before further processing, no more time limiting of 'old' segments
   no more dumb daynumber calculation for x weeks, also mixed filenumber
   extensions are possible. Verification by the first admin line of each
   segment (if possible)

Another bug report (don't know if its still fixed) mentions a problem with
packed segments. So pre-processing analyse routine also can check packed
segments, can unpack them and include the extracted files to the analyse
processing. And in the case a packed filename extension doesn't match to the
filename included (eg extension is z03 and tempted to believe that the file is 
of current date, but the segment is from day# 203 of year 2011 (probably last
x03 filename extension that matches x03 numbering schema - pack *.?03

similar routine I've implemented in some pointlist checker and processing tools
approx 15-20 years ago ...

              extension   filedate    admin line  valid  result   datetime
                                                  crc.ok
region56.003   current     current     current     yes   4     20131231210000
region56.361  not current not current not current  yes   1     20131224210700

extension calculation, upcoming Friday ->  003
  -> valid are all daynumbers between 361 and 003

datetime of upcoming Friday -> 20140103
  -> filedate in range 20140103 down to 20131227

admin line  segment for daynumber/date ->  003, 2014-01-03
  -> matches exactly

admin line crc matches nodelist-crc
  -> matches exactly

now you can programticly calculate a valid current segment
sort date desc ->  20131231, 20131224  -> use file 20131231