Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#35 Add date of last indexation

closed
Andre-Littoz
General (16)
5
2012-11-15
2012-10-21
Lukasz M
No

I run lxr indexation on regular basis. I would like to know if run was successful or not. Would it be possible to add date/time of last successful run somewhere on the page? Or have a separate page with history of successful/failed runs.

Discussion

  • Andre-Littoz
    Andre-Littoz
    2012-10-21

    The structure of the database is not compatible with an indexing history (in my opinion): there is provision for only one state of the references. Once genxref has begun its work, it should go to the end (unless something is wrong with the access rights). The only case of inconsistent DB is that latter error.

    I will experiment with a new table to contain a time stamp and put it somewhere in the header or footer.

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-01

    Transferred from "support request" to "feature request"

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-01

    • milestone: 122753 -->
     
  • Andre-Littoz
    Andre-Littoz
    2012-11-02

    • assigned_to: nobody --> ajlittoz
    • labels: --> General
     
  • Andre-Littoz
    Andre-Littoz
    2012-11-02

    Would it be right if the date of last indexation appears in the showconfig page?
    This way, the DB would be queried for this date only when that page is requested. Otherwise, if the date is printed in header/footer area, DB is queried for every page displayed. Note this is relatively unimportant because of the numerous other DB accesses. This could also be displayed in ident page where it is important to be aware of result relevancy.

     
  • Lukasz M
    Lukasz M
    2012-11-05

    I think it would be ok. Would there be a date per each version for given tree?

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-05

    Oh! I missed that point. Following your remark, I can add a field in lxr_status (which is associated with each base version of a file). With this implementation there would be a date with each file. There is no DB table associated with a "version" (because "versions" are rather virtual with some storage, such as CVS or Git).

    If date is in lxr_status, its retrieval is for free. The drawback is I can't provide it in identifier search (at least until I get a result).

    Also, when you index your tree incrementally, date for "file already indexed" does not change; date is updated only for changed files.

    Proposal:
    - directory listing: a new column with date of last indexation
    - file display: a line with date of last indexation

    What do you think?

     
  • Lukasz M
    Lukasz M
    2012-11-07

    Well, it's not really what I was thinking about. I was hoping to have last date of successful run of genxref (for each tree and version). Or last couple of runs - successful and failed ones. But its ok to not have this feature. :) Maybe later.

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-07

    What is the definition of "successful run"? or a "failed run"?

    Presently, there is nearly no status information returned. There are too many independent software layers with their own ways of reporting what they consider an error.

    For example, DB errors are "displayed" with an error message, but when it returns back to Perl, then to LXR, this information is lost!

    Globally, genxref is a dispatcher. If spawning an operation is OK, it considers it a "success", no matter how the dispatched task behaves.

    I have already implemented part of the "indexing time" column. It can give 2 pieces of information:
    1- whether the file really went through a parser (if not, column is blank; e.g. .txt files)
    2- whether the DB state is up-to-date with file state (if indexing time is BEFORE file revision time, then cross-references are not valid).

    For item 2, I'm having some trouble with svn. In the end, I could give a background color (light red) in directory listing for files which cross-references are not valid. I could also add a warning at the top of file content. And maybe, why not, have a background in identifier search for references in files which cross-refs are invalid.

    Would that be of any help for your concern?

     
  • Lukasz M
    Lukasz M
    2012-11-08

    I guess it's hard to determine successful run. I guess for me it is when genxref indexed/tried to index all the files. I think it could be even just finishing genxref (with or without error) because some servers may get overloaded and operation may not finish overnight. I think I would like to know that it is still running after 12h. :)
    For point 2, I think it could help. I have doubts about that background color - maybe just warning would be enough. Or maybe warning on directory listing page, that not all files are indexed? And I guess you will be able to determine per version if file needs indexing or not?

     
  • Lukasz M
    Lukasz M
    2012-11-08

    Actually "Or maybe warning on directory
    listing page, that not all files are indexed?" might not be so easy to do. Forget it. :)

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-08

    1- I understand that your main concern is determining if genxref is still running or not. I needed it also. That's why I changed genxref log so that work on a file is only one line long, with colours indicating status (or type of indexing). To quickly have an idea of where genxref is, directory name is repeated every so often the purple lines) so that it remains visible even with scrolling.

    Of course, if genxref is launched in the background (or on a remote computer as a batch or cron job), there is no associated terminal and you have no progress output. In that case, you can try 'top' utility (or at least 'ps', but it is not very user-friendly). On Windows, I think your choice is limited to the 'Task manager'.

    2- In my test implementation, I record the current time when reference collection terminates. Definition collection is supposed to have been done in a previous pass (the one handled by ctags). Since this previous pass with ctags is quite fast, I thought it was not necessary to log its completion time.

    This time-stamp is an individual file property. Consequently, you can use your browser to monitor indexing progress by refreshing a directory page. You'll see the changes in the "Last indexed" column.

    As presently implemented, different versions for a file have their own timestamp which are shown if they are OK. With a small smart test, I can make the difference between a file which cannot be indexed because there is no scanner for its language (marked as - in the 'last indexed' column) and a file modified since last genxref (marked as 'Not valid' since displaying the date would need a careful visual comparison between last modification and last indexed dates -- I chose this indication because the date is not important for me if it is stale).

    Visually, it is very fast to see which files will give questionable cross-reference results.

    3- As soon as I've checked the feature reliability, I'll also implement some flagging in identifier results where it is much more important to know indexing might be wrong when clicking on a line number (which might not jump on the identifier line!).

    4- I have not yet reindexed a kernel to measure the performance impact. On my small text cases, it seems negligible but duration is too short to create a real botherance.

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-08

    I've done the kernel test. Reported time by 'time ./genxref ...' is 2:40:12 instead of 2;39;40, i.e. it is the same within the uncertainty of the measurement. It is quite fun to monitor indexing progress in real time in directory display with a browser.

    I proceed with other tests, then I'll implement the warning flag in identifier search.

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-15

    Implemented in CVS - will be available in release 1.1

     
  • Andre-Littoz
    Andre-Littoz
    2012-11-15

    • status: open --> closed