lxr-0.11.1: glimpse/swish-e search fail

Help
2012-04-10
2013-01-02
  • conquistador
    conquistador
    2012-04-10

    Hi
    I setup lxr, index sources, startup apache2. It works: show sources, indent search, but not search script…
    When I try search something, I get "Search fail" in web page and

    /usr/bin/glimpse: unmatched '' (use \ to search for )

    Alsa doesn't work swish-e search…

    What's wrong, where is my mistake?

    Thank you

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-10

    Hello,

    Which LXR release are you using? See at bottom of page something like "0.11.1 LXR engine".
    Are you cross-referencing a personal source-tree or some publicly available one?

    What is your target string for search? The error message from glimpse suggests you have "reserved" characters in it (here "reserved" means characters used in glimpse-specific regular expressions).

    You can't have both glimpse and swish-e simultaneously active. To switch from one to the other, you must reconfigure lxr.conf and re-run genxref. Were there any errors during the glimpse ou swish-e indexing step? What do you mean by "swish-e doesn't work"? Any error printed pr just a blank screen?

    ajl

     
  • conquistador
    conquistador
    2012-04-10

    thanks for reply

    Which LXR release are you using? See at bottom of page something like "0.11.1 LXR engine".

    "This page was automatically generated by the 0.11 LXR engine. " - strange, I have downloaded lxr-0.11.1.tgz

    Are you cross-referencing a personal source-tree or some publicly available one?

    I cross-reference linux kernel 3.0.6 sources

    What is your target string for search?

    Any. For example: sk_buff
    http://lxr.sourceforge.net/cgi-bin/search?_filestring=&_string=sk_buff

    You can't have both glimpse and swish-e simultaneously active.

    I know it. I index it on two machines for testing. Now I am near glimse machine…

    Were there any errors during the glimpse ou swish-e indexing step?

    No, It worked fine. All done, and ident script work fine

    What do you mean by "swish-e doesn't work"? Any error printed pr just a blank screen?

    Switch-e print me, that my version isn't indexed (I'm not sure, machine at home). Glimpse write search page with message "Search fail"…
    This is my lxr.conf, may be it has mistake
    http://pastebin.com/KVf5A5vc

    PS: I have changed css way
    # CSS style sheet
    # , 'stylesheet' => 'templates/html/lxr.css'
    , 'stylesheet' => 'templates/lxr.css'

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-10

    "This page was automatically generated by the 0.11 LXR engine. " - strange, I have downloaded lxr-0.11.1.tgz

    Yep! I forgot to update LXRVersion variable, as I do frequently. To prevent that in the future, I modified the release script so that it automatically updates the version variable. I may even remove LXRversion.pm in the future. But you really downloaded 0.11.1.

    I recently worked a lot on "search" because it was unreliable and you did not get the error message from glimpse on screen (you had to extract them from the error log file, which is not user-friendly).

    You can't conclude that something is wrong with search for sk_buff in the demo on lxr.sourceforge.net because the demo works on LXR itself and there is no sk_buff in it. In this case "search fail" is correct because there is no occurrence ("0 occurrences found" printed in the results area). However, "search fail" may also be printed without anything else when there is an error. This is because the "historical" call on glimpse does not capture stderr where the error messages are printed. I patched it in 0.12-candidate to give more accurate error indication.

    If you give me an email@dress, I can send you an 0.12-beta version.
    Nota: in this 0.12 candidate, "case insentive" is now operational, which can limit the number of "false positives". sk_buff is too frequent in the kernel; I reach the 1000 limit, even with "case sensitive" which eliminates ASK_BUFFER for example. With ident, I get 35 declarations and 13579 references (3.1 kernel).

    No, It worked fine. All done, and ident script work fine

    search and ident use 2 different methods:
    - search is based on an external plain text search engine, e.g. glimpse or swish-e. The search engines build and use their own databases.
    - ident uses the internal LXR database prepared in 2 steps during genxref: first step to collect declarations (done by ctags), second step to enumerate all references (done by a set of Perl subroutines - very slow compared to ctags).
    Afterwards, ident is very fast because it is a simple database query, while search is rather lengthy because the engine looks up the full text.

    This is my lxr.conf, may be it has mistake

    I don't see obvious mistakes. Since you stored the search engines databases in /var/db (owned by root), are you sure that Apache and glimpse (which are launched under userid other than root) can access them? File permissions should be at least rwxr-xr-x. And maybe rwxrwxrwx since the search engine need to write its databases.

    PS: I have changed css way # CSS style sheet

    Shame on me! This is again a typo when I reorganised lxr.conf and added comments. I noticed it after I released the new version. I put a bug warning in lxr.sourceforge.net "Known bugs" page. I considered it was minor enough not to be worth a bug-fix release since user must edit lxr.conf anyway. Fix is already in CVS.

    PS: I'm tracking performance inefficiencies in LXR. I discovered that -reindexall results in an indexation (=genxref) time inflated by nearly 50% compared to initial indexation (when DB is empty). Can you tell me how long it takes to index your 3.0.6? What is your computer (processor, clock frequency, memory size)?
    Results for me: i5 3.4GHz 4GB
    initial run on 3.1: 3 hours 43 minutes (real time)
    Following run with -reindexall: 5 hours 47 minutes!!!
    (You get the figures with "time ./genxref …")

    ajl

     
  • conquistador
    conquistador
    2012-04-10

    You can't conclude that something is wrong with search for sk_buff in the demo on lxr.sourceforge.net because the demo works on LXR itself and there is no sk_buff in it.

    I understood it, I just show you page Error, like my :-)

    If you give me an email@dress, I can send you an 0.12-beta version.

    x.conquistador.x at "google mail com"
    Will I need to install all again? or just replace some scripts?

    Since you stored the search engines databases in /var/db (owned by root), are you sure that Apache and glimpse (which are launched under userid other than root) can access them?

    Yes, for reading access is valid. But when I index sources, I start glimpse from root
    "drwxr-xr-x 3 root root" for all path to glimpse_db. May be gentoo do it earlier.

    PS: I'm tracking performance inefficiencies in LXR.

    home: i7 870 (don't sure model), without overclocking, 4 Gb
    work: Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz, 6 Gb

    I will come home, and if I will not busy, start index && reindex with glimpse. At home I index kernel 3.3, I'm not sure, but first index is low 1 hour (40 min may be)… What indexer you interest?

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-10

    Beta 0.12 sent. Since I made a lot of changes, better to install all again but should be pretty easy with the install scripts. I think you can keep your present lxr.conf.

    I understand you're under Gentoo. Glad that indexing takes only 1 hour. A user reported that upgrading from Fedora 15 to 16, things turned catastrophic with same LXR version (see my 3:43:00 duration - I'm under Fedora 16, I remember a much faster indexing in previous distribution versions). In 0.12, I made ctags work directly on source files instead of making copies. But I'm disappointed: I get only a 10-15% improvement.

    I suggest using glimpse because it gives more detailed information than swish-e (glimpse shows the line where the string was found while swish-e only tells how many occurences there are in the file).

    If your total time is on the order of one hour, something is broken in Fedora. Depending on your result, I'll install a Gentoo as a virtual machine to have a look at that issue.

    Regards,
    ajl

     
  • conquistador
    conquistador
    2012-04-10

    Did you send my to  x.conquistador.x at gmail com?
    I don't see anything…

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-10

    Yes, at 15:35 UTC. No error return.

     
  • conquistador
    conquistador
    2012-04-10

    ok, recieved… thank you

     
  • conquistador
    conquistador
    2012-04-10

    I lied to you.
    First index on Intel(R) Core(TM) i7 CPU 870  @ 2.93GHz, 4Gb, kernel 3.3 -> 1h30min, by 0.12.1, with kde + firefox :-)
    Want to ask you about primary problem, did you fix it? or only move perl error to web page from apache log?

     
  • conquistador
    conquistador
    2012-04-11

    Sorry, but I can't use 0.12.1 version, when I start it I meet some problem:
    - css file search in lxr dir, not template
    - source files doesn't open, dir content is visible, but file view doen't possible
    http://pastebin.com/6YSi52Ys

    may be it my mistake, may be version is to raw… don't speed.. I can use 0.11 version without search yet…

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-13

    Hi,
    1- I don't understand the first problem: do you mean you had to copy lxr.css from templates/ to the lxr root directory?
    2- It looks to me like an Apache configuration problem: source, diff, ident and search are not considered as Perl (CGI) scripts. Have you copied lxrconf.d/apache-lxrserver.conf into /etc/http/conf.d/ (or the equivalent for Gentoo)? If you did not copy the file, you cannot get Perl activation.

    ajl

    PS: I you are not at ease with English, I can also read Spanish, French and German.

     
  • conquistador
    conquistador
    2012-04-13

    I have answered at your mail, send you some configs and version info

    Source , Ident scripts work fine… problem only in search still

     
  • Andre-Littoz
    Andre-Littoz
    2012-04-14

    After a lot of private mail exchanges, it seems the error message

    /usr/bin/glimpse: unmatched '[', ']' (use \[, \] to search for [, ])
    

    appears with some Linux distributions (e.g. Gentoo) and not with others (e.g. Fedora).

    This means there is a non consistent implementation of glimpse across the distributions. Consequently, I open a bug to make a note if people meet the same difficulty.

     
  • conquistador
    conquistador
    2012-04-24

    0.12.1 works fine, thank you