lxr-developer — Don't use, active developers only

[Lxr-dev] [ lxr-Bugs-506627 ] Excessive memory usage

[<no...@so...>]

Bugs item #506627, was opened at 2002-01-21 12:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

Category: genxref
Group: current cvs
Status: Closed
Resolution: Fixed
Priority: 4
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Malcolm Box (mbox)
Summary: Excessive memory usage

Initial Comment:
LXR uses much more memory than is required. I have a 
large binary with debugging symbols (80+M) in my 
source tree and attempting to index it I get the 
following error.

Out of memory during "large" request for 134221824 
bytes, total sbrk() is 203704320 bytes at 
lib/LXR/Lang.pm line 42.

Note the first value is equal to 128M. A quick look at 
the source code shows the offending line below:

# Try to see if it's a script
$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;

And the offending function from Plain.pm:

sub getfile {
        my ($self, $filename, $release) = @_;
        my ($buffer);
        local ($/) = undef;

        open(FILE, $self->toreal($filename, $release)) 
|| return undef;
        $buffer = <FILE>;
        close(FILE);
        return $buffer;
}

Reading the whole file into memory to determine if it 
is a shell script is really excessive. Surely the 
first line should suffice. File::Type may do a better 
job, but it operates only on a filename, and I'm not 
sure about portability.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2002-01-23 10:15

Message:
Logged In: NO 

> Keeping binaries in a source tree is rarely a good thing.

Preaching to the choir...
But the fact of the matter is that large files often do end
up in source trees (trace files, log files, docs ...) just
because it helps for sharing and organization.

Even with this bug fixed, memory usage seems excessive. I
left LXR running for about a day, and found it had slowed to
a crawl because it was swapping constantly. Each perl
process was using about 200 M. It crashed the following day
when it ran out of memory.  I don't know what caused this
yet (I just started it up again).

----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2002-01-22 23:58

Message:
Logged In: YES 
user_id=215386

I suppose one answer would be "Don't do that then" :-) 
Keeping binaries in a source tree is rarely a good thing.

In terms of fixing this, the easiest way would be to use the
Files::getfilehandle rather than Files::getfile to read the
first line.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

[Lxr-dev] [ lxr-Feature Requests-489929 ] function index (speedbar)

[<no...@so...>]

Feature Requests item #489929, was opened at 2001-12-06 09:53
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=489929&group_id=27350

Category: Browsing Interface
Group: None
Status: Open
Priority: 5
Submitted By: David Oleszkiewicz (poppinfresh)
Assigned to: Nobody/Anonymous (nobody)
Summary: function index (speedbar)

Initial Comment:
I use the speedbar in xemacs alot to help be jump 
straight down to a function in a source file.  Well i 
actually map it to the mouse...but it would be nice 
to have a speebar (html frame) on the the left or 
right hand side of the source file which has links to 
all of the functions declared in a source file.  
presumably this information is already parsed in a 
manner that you know it is a function def.  this 
could be ignored in header files or contain the index 
of typedefs and #defines or something.  


----------------------------------------------------------------------

>Comment By: Malcolm Box (mbox)
Date: 2002-01-23 08:08

Message:
Logged In: YES 
user_id=215386

Wouldn't be hard.  The sql to find all the symbols defined
in a file is:

select s.symname, i.line, d.declaration from indexes i,
symbols s, declarations d, files f where s.symi
d=i.symid and i.type = d.declid and i.fileid=f.fileid and
f.filename = "filename";

From there, it should be easy to spin some HTML and links
round the results and shove it in a sidebar.  Presumably
this would be linked off the source script, so that it would
first generate the source listing, and then add the sidebar
panel.


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=489929&group_id=27350

[Lxr-dev] [ lxr-Feature Requests-466170 ] Expanded CSS support

[<no...@so...>]

Feature Requests item #466170, was opened at 2001-09-28 12:42
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=466170&group_id=27350

Category: Browsing Interface
Group: None
>Status: Closed
Priority: 5
Submitted By: Joseph Wilhelm (tarken)
Assigned to: Nobody/Anonymous (nobody)
Summary: Expanded CSS support

Initial Comment:
I had just one small requested addition to the CSS. It 
would be nice to see, with the blame-annotated source, 
alternating colors the way Bonsai does; switching 
colors with each different blame. Just a small 
cosmetic addition. :)

----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2001-10-31 00:41

Message:
Logged In: YES 
user_id=215386

What blame-annotated source?  I don't know of any blame
annotated output from LXR, only from cvsweb.

Could you explain what you mean?

Malcolm

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=466170&group_id=27350

[Lxr-dev] [ lxr-Feature Requests-433936 ] Remove dependance on glimpse

[<no...@so...>]

Feature Requests item #433936, was opened at 2001-06-17 08:54
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=433936&group_id=27350

Category: General
Group: Next Release
>Status: Closed
Priority: 5
Submitted By: Malcolm Box (mbox)
Assigned to: Arne Georg Gleditsch (argggh)
Summary: Remove dependance on glimpse

Initial Comment:
The current free text search interface relies on the
glimpse package, which is not Free software.  Thus we
should try to move to a free work-alike.

Suggestions (from Neil Slater):

I know mngosearch has been suggested before now:

http://search.mnogo.ru/

...as has htdig:

http://www.htdig.org/

However, whilst trying to locate the mngo URL, I
chanced across the
following page which also lists a number of other free
search/indexing
engines:

http://twiki.org/cgi-bin/view/Codev/SearchAttachments


Note that there is slightly more to this work than just
ripping out glimpse, although that would be a start. 
The current code assumes that the files to be indexed
are stored in a plain directory tree, and therefore
will not work with the CVS or other File backend.  This
would really need to be fixed.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=433936&group_id=27350

[Lxr-dev] [ lxr-Bugs-481573 ] requires non-free software for searching

[<no...@so...>]

Bugs item #481573, was opened at 2001-11-13 20:05
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=481573&group_id=27350

Category: None
Group: None
>Status: Closed
>Resolution: Fixed
>Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Arne Georg Gleditsch (argggh)
Summary: requires non-free software for searching

Initial Comment:
On your front page you suggest Glimpse, which is
horridly non-free:
http://www.arco.de/~kj/harvest/glimpse-license-status

This program is GPL, so a user might assume that it's
dependancies are GPL. 

Please provide hooks for Swish-E http://swish-e.org/ or
Swish++
http://homepage.mac.com/pauljlucas/software/swish/
which are GPL replacements.
If I am in error and this feature allready exists in
lxr, please change the content of http://lxr.linux.no/
to suggest Swish-E or Swish++.


I might note that I caught this as a part of our
department's implementation of Bugzilla (bonsai uses
lxr) which is a mission-critical application for
development here, and now that we know that Glimpse has
*never* been in the public domain (the source we found
had a misleading copyright file) we need to find a
replacement or pay up before the audit rolls around.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-12-22 03:16

Message:
Logged In: NO 

The thread at the following URL provides some background 
information on the possibility of using swish++

http://groups.yahoo.com/group/swish/message/538

Cheers,

Neil.



----------------------------------------------------------------------

Comment By: Rusty Carruth (rustyc)
Date: 2001-11-14 06:25

Message:
Logged In: YES 
user_id=215914

There *is* a version of glimpse (I think around 4.0) which
does 
not require a fee.  This is the version we are using.  A
temporary workaround would be to grab that version and use
it while 
'someone' is fixing lxr to use the free alternative.

(We found that version of glimpse by looking around on the
'net till we found it.)

rc


----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2001-11-14 00:06

Message:
Logged In: YES 
user_id=215386

I agree, depending on non-free software for a tool like LXR
is not a good thing.  Moving over to one of the free
alternatives is on the to-do list.

If you wanted to go ahead and provide the hooks for a
replacement, I'd be very happy to incorporate the patch in
the LXR.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=481573&group_id=27350

[Lxr-dev] Re: [Lxr-commits] CVS: lxr/lib/LXR Lang.pm,1.25,1.26

[Jan-Benedict G. <jb...@lu...>]

On Wed, 2002-01-23 23:22:47 +0900, Malcolm Box <ma...@br...>
wrote in message <3C4...@br...>:

Hi,

> Jan-Benedict Glaw wrote:
> >Spoken C'ish, there's an array of structs in lxr.conf (sorry,
> >I'm not Perl-enabled...) containing language descriptions. This
> >includes some possible filename extensions which are almost
> >reasonable. Why don't we add a struct member called "interpreters"
> >containing a list of possible interpreters, if there's not a match
> >for the filename extension.
> >
> Do you mean that this would be a seperate meta-language called 
> 'interpreters', or that the perl/bash etc entries would also include 
> their interpreter strings?

I would do the later one. I think it's quite easier to do it
that way around...

> >Lang.pm does this exceptional for Perl, I'd like to see this behavior
> >to be a bit more flexible by scanning over thise things there in
> >lxr.conf. There's no need for a perl script to end up in .pl, equally
> >there's no need for s shell script to end up in .sh. So why don't we
> >generally also test for a shell bang (if there's no extension)?
> >
> One way to do this of course is simply to extend Lang.pm - it's pretty 
> trivial, though not exactly flexible.
> 
> I think the best way to do it is to have an 'interpreters' entry in 
> lxr.conf that maps between potential #! strings and languages.  Then to 

That's fine with me, too:-)

> get the right info, just look the language name back up in the filetype 
> mapping.  This avoids having to specify things like the module or 
> tabwidth twice, without the overhead of looking at the first line of the 
> file if the filename mapping works out.
> 
> In fact, I'm so inspired I'll even code it.  How's the for volunteers?

Oh yes, please, do it. I like it, but I can't do it. I've looked
at Perl several times, I always failed. My braindumps look more
like C...

MfG, JBG

-- 
Jan-Benedict Glaw   .   jb...@lu...   .   +49-172-7608481
	 -- New APT-Proxy written in shell script --
	   http://lug-owl.de/~jbglaw/software/ap2/

[Lxr-dev] Re: [Lxr-commits] CVS: lxr/lib/LXR Lang.pm,1.25,1.26

[Malcolm B. <ma...@br...>]

Hi,



Jan-Benedict Glaw wrote:

>Please, take Lang.pm and read some further lines. There, we fetch
>a (possible) shell bang and test it against the occurence of "perl".
>Bad.
>
Yep, you're right, this is special-casing for perl.  If anyone fancies 
coding up a patch to do what JBG suggests below then I'll be happy to 
add it.

>Spoken C'ish, there's an array of structs in lxr.conf (sorry,
>I'm not Perl-enabled...) containing language descriptions. This
>includes some possible filename extensions which are almost
>reasonable. Why don't we add a struct member called "interpreters"
>containing a list of possible interpreters, if there's not a match
>for the filename extension.
>
Do you mean that this would be a seperate meta-language called 
'interpreters', or that the perl/bash etc entries would also include 
their interpreter strings?

>Lang.pm does this exceptional for Perl, I'd like to see this behavior
>to be a bit more flexible by scanning over thise things there in
>lxr.conf. There's no need for a perl script to end up in .pl, equally
>there's no need for s shell script to end up in .sh. So why don't we
>generally also test for a shell bang (if there's no extension)?
>
One way to do this of course is simply to extend Lang.pm - it's pretty 
trivial, though not exactly flexible.

I think the best way to do it is to have an 'interpreters' entry in 
lxr.conf that maps between potential #! strings and languages.  Then to 
get the right info, just look the language name back up in the filetype 
mapping.  This avoids having to specify things like the module or 
tabwidth twice, without the overhead of looking at the first line of the 
file if the filename mapping works out.

In fact, I'm so inspired I'll even code it.  How's the for volunteers?

Malcolm

Re: [Lxr-dev] Permission problems with Postgres?

[Jan-Benedict G. <jb...@lu...>]

On Wed, 2002-01-23 22:26:18 +0900, Malcolm Box <ma...@br...>
wrote in message <3C4...@br...>:

> However, on doing the install, there seems to be a problem with 
> permissions.  Specifically, I get permission denied messages when the 
> lxr user attempts to insert any values into the database.  Now I'm not 
> familiar with postgres, so does anyone else experience this, or am I 
> just doing something wrong?
> 
> Here's what I did:
> 
> createuser lxr
> - I answered "yes" to the permission questions to give lxr lots of rights
> createdb -U lxr lxr
> psql -U lxr
> \i initdb-postgres

Basically, it works for me. I'm regulary using PostgreSQL, I don't
exactly like MySQL...

What I did...

I've already got a user jbglaw, capable of creating databases but
not capable of creating new users.

As jbglaw, I did a "createdb lxr", followed by "psql lxr" and therein
a "\i initdb-postgres". This init scripts seems to work for me. Lxr is
running under apache, which uses user "www-data" (which is also a
user in PostgreSQL for me).

So, you should additionally create a database user called like the
user used by your apache and everything should be fine, because the
initdb script already has grant statements.

You *can't* ./genxref (with the current solution) as user www-data,
read: you can't index new sources while you're your webserver's user.

But that's fine with me. jbglaw created the database, granting
select rights to public. www-data *is* public, so it can use the
database "read only", but that needs to be enough.

Nobody else than the database's creator should be granted write
(read: UPDATE, DELETE or INSERT) right to the database except


Last famous words: The current initdb-postgres is okay.

MfG, JBG

-- 
Jan-Benedict Glaw   .   jb...@lu...   .   +49-172-7608481
	 -- New APT-Proxy written in shell script --
	   http://lug-owl.de/~jbglaw/software/ap2/

[Lxr-dev] Permission problems with Postgres?

[Malcolm B. <ma...@br...>]

Hi folks,

I've just tried to install Postgres for use with lxr, since I normally 
use MySQL and thought I ought to test both, or at least the install 
instructions.

However, on doing the install, there seems to be a problem with 
permissions.  Specifically, I get permission denied messages when the 
lxr user attempts to insert any values into the database.  Now I'm not 
familiar with postgres, so does anyone else experience this, or am I 
just doing something wrong?

Here's what I did:

createuser lxr
 - I answered "yes" to the permission questions to give lxr lots of rights
createdb -U lxr lxr
psql -U lxr
\i initdb-postgres

Then I tried to run genxref, and it blew up when first trying to insert 
into a table.  Going into psql and looking at the table permissions, it 
shows lxr as the owner for the tables, but only public read access in 
the permissions.  This appears to conflict with the documentation, which 
claims that the owner always has all permissions.

I then did a "grant all" on all the tables to lxr, and after that 
genxref worked fine.  So should I update the initdb-postgres script to 
add the relevant grant statements at the end?  What do others of you who 
use Postgres do - or is it only my setup which is strange.  If it 
matters I'm using Postgres 7.0.3.

Cheers,

Malcolm

[Lxr-dev] [ lxr-Bugs-506845 ] missing variable in lxr.conf

[<no...@so...>]

Bugs item #506845, was opened at 2002-01-21 23:43
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506845&group_id=27350

Category: Browsing
Group: v0.9
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Anton Todorov (antod)
Assigned to: Nobody/Anonymous (nobody)
Summary: missing variable in lxr.conf

Initial Comment:
Hi,
I am using lxr in plain file mode, mysql as database.
When I enabled glimpse support in configuration found that is 
missing variable "dbdir" in lxr.conf file.  So find and search 
modules do not work. I look in lxr v0.3 configuration file and 
found what is the right value for this variable andd add it in 
baseurl configuration section of lxr.conf. Now find and search 
modules are working fine. Here is the added line to lxr.conf:
'dbdir' => '/path/to/glimpse/database',
($v is defined variable in the same configuration block)

Best Regards,
Anton Todorov


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506845&group_id=27350

[Lxr-dev] [ lxr-Bugs-506627 ] Excessive memory usage

[<no...@so...>]

Bugs item #506627, was opened at 2002-01-21 12:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

Category: genxref
Group: current cvs
>Status: Closed
>Resolution: Fixed
Priority: 4
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Malcolm Box (mbox)
Summary: Excessive memory usage

Initial Comment:
LXR uses much more memory than is required. I have a 
large binary with debugging symbols (80+M) in my 
source tree and attempting to index it I get the 
following error.

Out of memory during "large" request for 134221824 
bytes, total sbrk() is 203704320 bytes at 
lib/LXR/Lang.pm line 42.

Note the first value is equal to 128M. A quick look at 
the source code shows the offending line below:

# Try to see if it's a script
$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;

And the offending function from Plain.pm:

sub getfile {
        my ($self, $filename, $release) = @_;
        my ($buffer);
        local ($/) = undef;

        open(FILE, $self->toreal($filename, $release)) 
|| return undef;
        $buffer = <FILE>;
        close(FILE);
        return $buffer;
}

Reading the whole file into memory to determine if it 
is a shell script is really excessive. Surely the 
first line should suffice. File::Type may do a better 
job, but it operates only on a filename, and I'm not 
sure about portability.

----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2002-01-22 23:58

Message:
Logged In: YES 
user_id=215386

I suppose one answer would be "Don't do that then" :-) 
Keeping binaries in a source tree is rarely a good thing.

In terms of fixing this, the easiest way would be to use the
Files::getfilehandle rather than Files::getfile to read the
first line.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

[Lxr-dev] [ lxr-Bugs-506627 ] Excessive memory usage

[<no...@so...>]

Bugs item #506627, was opened at 2002-01-21 12:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

Category: genxref
>Group: current cvs
Status: Open
Resolution: None
>Priority: 4
Submitted By: Nobody/Anonymous (nobody)
>Assigned to: Malcolm Box (mbox)
Summary: Excessive memory usage

Initial Comment:
LXR uses much more memory than is required. I have a 
large binary with debugging symbols (80+M) in my 
source tree and attempting to index it I get the 
following error.

Out of memory during "large" request for 134221824 
bytes, total sbrk() is 203704320 bytes at 
lib/LXR/Lang.pm line 42.

Note the first value is equal to 128M. A quick look at 
the source code shows the offending line below:

# Try to see if it's a script
$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;

And the offending function from Plain.pm:

sub getfile {
        my ($self, $filename, $release) = @_;
        my ($buffer);
        local ($/) = undef;

        open(FILE, $self->toreal($filename, $release)) 
|| return undef;
        $buffer = <FILE>;
        close(FILE);
        return $buffer;
}

Reading the whole file into memory to determine if it 
is a shell script is really excessive. Surely the 
first line should suffice. File::Type may do a better 
job, but it operates only on a filename, and I'm not 
sure about portability.

----------------------------------------------------------------------

>Comment By: Malcolm Box (mbox)
Date: 2002-01-22 23:58

Message:
Logged In: YES 
user_id=215386

I suppose one answer would be "Don't do that then" :-) 
Keeping binaries in a source tree is rarely a good thing.

In terms of fixing this, the easiest way would be to use the
Files::getfilehandle rather than Files::getfile to read the
first line.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

[Lxr-dev] [ lxr-Patches-507401 ] Spec file for RPMs

[<no...@so...>]

Patches item #507401, was opened at 2002-01-22 23:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390119&aid=507401&group_id=27350

Category: Other
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Malcolm Box (mbox)
Assigned to: Malcolm Box (mbox)
Summary: Spec file for RPMs

Initial Comment:
Use the spec from Nicholas posted on the lxr-general
mailing list to create RPMs for lxr.  This should help
ease the install problems people experience.

Patch is at
http://sourceforge.net/mailarchive/forum.php?thread_id=48430&forum_id=1734

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390119&aid=507401&group_id=27350

[Lxr-dev] [ lxr-Bugs-506845 ] missing variable in lxr.conf

[<no...@so...>]

Bugs item #506845, was opened at 2002-01-21 23:43
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506845&group_id=27350

Category: Browsing
Group: v0.9
Status: Open
Resolution: None
Priority: 5
Submitted By: Anton Todorov (antod)
Assigned to: Nobody/Anonymous (nobody)
Summary: missing variable in lxr.conf

Initial Comment:
Hi,
I am using lxr in plain file mode, mysql as database.
When I enabled glimpse support in configuration found that is 
missing variable "dbdir" in lxr.conf file.  So find and search 
modules do not work. I look in lxr v0.3 configuration file and 
found what is the right value for this variable andd add it in 
baseurl configuration section of lxr.conf. Now find and search 
modules are working fine. Here is the added line to lxr.conf:
'dbdir' => '/path/to/glimpse/database',
($v is defined variable in the same configuration block)

Best Regards,
Anton Todorov


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506845&group_id=27350

[Lxr-dev] Re: Got LXR 0.9 working on windows!

[Shree K. <sh...@pr...>]

Hi Malcolm,
---------------------------------------------------------------------
> 
> Were there any special config steps you had to take, or was it literally
> plug'n'play?  
There were no special config steps. But contrary to my previous mail,
one 
small change was required to 'generic.pm'. The windows version of ctags
does not recognize unix style directory separators. Hence the following
change:

---
 if ($config->ectagsbin) {
       # Windows specific patch - ctags.exe expects \tmp\lxrtemp...
instead of /tmp/lxrtemp.... 
       # ie replace forward slash with backslash to account for Windows
filenaming convention
       $path =~ s/\//\\/;
       open(CTAGS, join(" ", $config->ectagsbin,
                                        $self->ectagsopts,
                                        "--excmd=number",
                                        "--language-force=$langforce",
                                        "-f", "-", 
---

If the cygwin version of ctags were used, no changes would be required,
however.

Before running genxref, the user must ensure that the drive [c:, d:,
etc] from genxref is executed
must have a \tmp directory

---------------------------------------------------------------------
> If there were, I'm sure a brief posting of them would be
> much appreciated.
> 
But I still don't have the CVS part working. What happened is that I
dabbled with NTFS file
permissions and now LXR is not able to read the some files and
directories.  So do you think it
is advisable to post it now? And to which forum should I post the
developments?

---------------------------------------------------------------------
> > I have also added an additional feature - regular expression based
> > identifier search.
> 
> Sounds cool. Any chance of patches?

Sure. I have added the code currently only for MySQL. But I have not
used the regular expression based
query facilities available in MySQL. Instead I get all identifiers and
match with the supplied regexp.
I have found the facility were useful indeed! One bad thing is that I
have not yet made it conformant
with CSS use. Your opinion ?

--------------------------------------------------------------------
One more feature that would be helpful is a GUI frontend for lxr.conf.

This would prevent a lot of frustration due to non-existent/ wrong path, 
inappropriate permission on files, etc.

Any takers ?


Cheers,

Shree Kumar

[Lxr-dev] [ lxr-Bugs-506627 ] Excessive memory usage

[<no...@so...>]

Bugs item #506627, was opened at 2002-01-21 12:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

Category: genxref
Group: v0.9
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Excessive memory usage

Initial Comment:
LXR uses much more memory than is required. I have a 
large binary with debugging symbols (80+M) in my 
source tree and attempting to index it I get the 
following error.

Out of memory during "large" request for 134221824 
bytes, total sbrk() is 203704320 bytes at 
lib/LXR/Lang.pm line 42.

Note the first value is equal to 128M. A quick look at 
the source code shows the offending line below:

# Try to see if it's a script
$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;

And the offending function from Plain.pm:

sub getfile {
        my ($self, $filename, $release) = @_;
        my ($buffer);
        local ($/) = undef;

        open(FILE, $self->toreal($filename, $release)) 
|| return undef;
        $buffer = <FILE>;
        close(FILE);
        return $buffer;
}

Reading the whole file into memory to determine if it 
is a shell script is really excessive. Surely the 
first line should suffice. File::Type may do a better 
job, but it operates only on a filename, and I'm not 
sure about portability.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

[Lxr-dev] Re: Got LXR 0.9 working on windows!

[Malcolm B. <ma...@br...>]

Hi Shree,

Shree Kumar wrote:
> I have got LXR 0.9 working on Windows 2000.

That's great - thanks for letting me know.

> I used the following :
> 
> 1. mod_perl + Apache 1.3.20 bundle from
> ftp://theoryx5.uwinnipeg.ca/pub/other/perl-win32-bin-0.7.exe
>   [this mod_perl is based on ActiveState Perl]
> 2. MySQL
> 
> No changes to the LXR 0.9 code were required.

Were there any special config steps you had to take, or was it literally
plug'n'play?  If there were, I'm sure a brief posting of them would be
much appreciated.

> I have also added an additional feature - regular expression based
> identifier search.

Sounds cool. Any chance of patches?

Cheers,

Malcolm

[Lxr-dev] Re: [Lxr-general] LXR 09 and fortran

[Malcolm B. <ma...@br...>]

Hi Paul,

Paul Ogranovich wrote:
>     The question I have is about possibility of configuring LXR for
> processing Fortran source. If I am not mistaken, I remember reading
> _somewhere_ that version 09 either supports Fortran already or can
> be configured to support it... Is this correct ? If the answer is "yes",
> how do I configure LXR ?

The answer is that theoretically all you need to support Fortran is in
place, but the actual setup and testing of it has not been done.

To explain in more detail, lxr uses the Exuberant Ctags program to find
identifiers in source files.  Ctags (http://sf.net/projects/ctags) does
support Fortran, so that part is fine.  Lxr has a perl module to
interface to ctags called Generic.pm that is capable of handling every
language that ctags can handle.  Generic.pm is configured via the
generic.conf file in lib/LXR/Lang with details of how to deal with the
output of ctags for different languages.  However, the relevant entries
for Fortran have not yet been written into this file.

So to make Fortran work with lxr, you would need to fill in the details
in generic.conf.  The required details are as follows:

Fill in the 'langmap' variable.  This is a hash with keys being language
names and values being hashes.

You would need to fill in the Fortran hash as follows:

'Fortran' => 	{
	'reserved' => [ array of reserved words in Fortran ],
	'spec' => [ Set of regexps to parse Fortran (see below) ],
	'typemap' => { Hash mapping ctags type codes to strings, see ctags man
page for details on what type codes it uses for Fortran, already filled
in },
	'langid' => '4'
}

The typemap values have already been filled in in the latest CVS
version.
'spec' is the slightly tricky one to write.  The format is:

'token type', 'stat regexp', 'end regexp'.

The token type is one of "atom", "string", "include" and "comment".  You
can ignore "atom", and the others are reasonably self-explanatory.  For
example, if strings are deliminated with double quotes, the entry would
be:

'string', '"', '"'

and for C style comments the entry is:

'comment', '/\*', '\*/'

You can have as many of each token type as needed to find all the
occurances (e.g. C++ has two comment entries for C and C++ style
comments).

The "include" token regexps should match the entire include directive,
not just the included filename.

Once generic.conf has been filled in, the only remaining configuration
required is to associate the Fortran files with Generic.pm.  This is
done in lxr.conf, simply uncomment the existing line.

That should be it - Fortran should now work for you.  If you do get this
working, please could you send the relevant patches to lxr-developer?

Cheers,

Malcolm
> 
> Thanks for your help,
> 
> Paul Ogranovich.
> (pau...@ms...)
> 
> _______________________________________________
> Lxr-general mailing list
> Lxr...@li...
> https://lists.sourceforge.net/lists/listinfo/lxr-general

[Lxr-dev] [ lxr-Bugs-500773 ] no revisions are available on any page

[<no...@so...>]

Bugs item #500773, was opened at 2002-01-08 03:30
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=500773&group_id=27350

Category: Browsing
Group: v0.9
Status: Open
Resolution: None
Priority: 5
Submitted By: Bartlomiej Pater (noster)
Assigned to: Nobody/Anonymous (nobody)
Summary: no revisions are available on any page

Initial Comment:
hello,

I have problem with 0.9 version of lxr. After I have finally came to a solution with how to make it to read sources from CVS (I have to adopt config file - see below) I am unable to browse it. When I look into database created by genxref (see attachment) it seems it finds the files and their versions correctly. But when I try to browse it I can only see some 'head' tag/version. If I put versions in "range" variable in the config file it is ok, and I can do everything, but I need CVS support to know the version numbers everytime they change.

I have also tried version from CVS [about week ago] but with the same effect.

My configuration:
- gnu-linux-i386pc [debian sid]
- apache 1.3.22
- mysql 3.23.46
- perl 5.6.1 and 5.005 [v6.3 whatever it means - I'm not perl guru]
- libdbi-perl 1.20
- libdbd-mysql-perl 1.2216

I have tried it also at different machine but with very similiar configuration and with the same effect.


----------------------------------------------------------------------

>Comment By: Bartlomiej Pater (noster)
Date: 2002-01-08 03:47

Message:
Logged In: YES 
user_id=22605

this is the way I had to change the config to make it work
at all [well, part of it]:

'range'   => sub { return ($files->allreleases("/"),
$files->allrevisions("/"))                                 
      },


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=500773&group_id=27350

[Lxr-dev] [ lxr-Bugs-500773 ] no revisions are available on any page

[<no...@so...>]

Bugs item #500773, was opened at 2002-01-08 03:30
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=500773&group_id=27350

Category: Browsing
Group: v0.9
Status: Open
Resolution: None
Priority: 5
Submitted By: Bartlomiej Pater (noster)
Assigned to: Nobody/Anonymous (nobody)
Summary: no revisions are available on any page

Initial Comment:
hello,

I have problem with 0.9 version of lxr. After I have finally came to a solution with how to make it to read sources from CVS (I have to adopt config file - see below) I am unable to browse it. When I look into database created by genxref (see attachment) it seems it finds the files and their versions correctly. But when I try to browse it I can only see some 'head' tag/version. If I put versions in "range" variable in the config file it is ok, and I can do everything, but I need CVS support to know the version numbers everytime they change.

I have also tried version from CVS [about week ago] but with the same effect.

My configuration:
- gnu-linux-i386pc [debian sid]
- apache 1.3.22
- mysql 3.23.46
- perl 5.6.1 and 5.005 [v6.3 whatever it means - I'm not perl guru]
- libdbi-perl 1.20
- libdbd-mysql-perl 1.2216

I have tried it also at different machine but with very similiar configuration and with the same effect.


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=500773&group_id=27350

[Lxr-dev] [ lxr-Feature Requests-481579 ] generate static pages

[<no...@so...>]

Feature Requests item #481579, was opened at 2001-11-13 21:00
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=481579&group_id=27350

Category: General
Group: None
Status: Open
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: generate static pages

Initial Comment:
Hi:

Although I'm still battling to make it work, it's a
great project. I just have a simple wish:

- Give lxr an option to generate static pages for a
piece of code. Take my case: I frequently have to study
some pieces of code. lxr can speed up a lot my work,
but I don't have access to a web server. What I would
like to do (and generally do with html manuals) is to
burn a cd with the code cross referenced, and be able
to browse it at leisure. I already tried to do that
with a wget -r of a branch of the linux kernel, but it
didn't work.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-12-23 21:18

Message:
Logged In: NO 

I think you will find GLOBAL useful. GLOBAL is a tags system without www interface. It also can generate 
static HTML. Follow this link: www.tamacom.com/global


----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2001-11-28 05:35

Message:
Logged In: YES 
user_id=215386

What was it specifically that didn't work for you?

The problem might be that in addition to each file of source
(the output of the source program) you also need the ident
output for each of the symbols used.  Which might be several
thousand.  If the symbols then are referenced in file
outside the tree you're interested in then the spider could
get confused.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=481579&group_id=27350

[Lxr-dev] [ lxr-Bugs-481573 ] requires non-free software for searching

[<no...@so...>]

Bugs item #481573, was opened at 2001-11-13 20:05
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=481573&group_id=27350

Category: None
Group: None
Status: Open
Resolution: None
Priority: 9
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Arne Georg Gleditsch (argggh)
Summary: requires non-free software for searching

Initial Comment:
On your front page you suggest Glimpse, which is
horridly non-free:
http://www.arco.de/~kj/harvest/glimpse-license-status

This program is GPL, so a user might assume that it's
dependancies are GPL. 

Please provide hooks for Swish-E http://swish-e.org/ or
Swish++
http://homepage.mac.com/pauljlucas/software/swish/
which are GPL replacements.
If I am in error and this feature allready exists in
lxr, please change the content of http://lxr.linux.no/
to suggest Swish-E or Swish++.


I might note that I caught this as a part of our
department's implementation of Bugzilla (bonsai uses
lxr) which is a mission-critical application for
development here, and now that we know that Glimpse has
*never* been in the public domain (the source we found
had a misleading copyright file) we need to find a
replacement or pay up before the audit rolls around.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-12-22 03:16

Message:
Logged In: NO 

The thread at the following URL provides some background 
information on the possibility of using swish++

http://groups.yahoo.com/group/swish/message/538

Cheers,

Neil.



----------------------------------------------------------------------

Comment By: Rusty Carruth (rustyc)
Date: 2001-11-14 06:25

Message:
Logged In: YES 
user_id=215914

There *is* a version of glimpse (I think around 4.0) which
does 
not require a fee.  This is the version we are using.  A
temporary workaround would be to grab that version and use
it while 
'someone' is fixing lxr to use the free alternative.

(We found that version of glimpse by looking around on the
'net till we found it.)

rc


----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2001-11-14 00:06

Message:
Logged In: YES 
user_id=215386

I agree, depending on non-free software for a tool like LXR
is not a good thing.  Moving over to one of the free
alternatives is on the to-do list.

If you wanted to go ahead and provide the hooks for a
replacement, I'd be very happy to incorporate the patch in
the LXR.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=481573&group_id=27350

Re: [Lxr-dev] Swish-E

[Neil S. <ns...@ci...>]

I meant to post this to the list, sorry...

----- Original Message -----
From: "Neil Salter" <ns...@ci...>
To: "Per Kristian Gjermshus" <pk...@ne...>
Sent: Thursday, December 06, 2001 3:12 PM
Subject: Re: [Lxr-dev] Swish-E


> Oooops. I've only just found this thread.
>
> There's a C++ rewrite of Swish-E called Swish++. It's free (speech and
> beer), and uses the C++ Standard Library (along with all of its amazing
> containers) for speed. It seems to have been written with optimisation in
> mind, though I've not actually tried it yet.
>
> At present it has modules to index plain text, mail & news messages, and
> something else I can't remember. Right now it won't index C++, which is
what
> I'm interested in, but I'm currently looking into how easy/hard it would
be
> to make it do this.
>
> Has anyone else tried this yet?
>
> Website at: http://homepage.mac.com/pauljlucas/software/swish/
>
> Cheers,
>
> Neil.
>
> ----- Original Message -----
> From: "Per Kristian Gjermshus" <pk...@ne...>
> To: "Arne Georg Gleditsch" <ar...@li...>
> Cc: <lxr...@li...>
> Sent: Thursday, November 22, 2001 8:11 AM
> Subject: Re: [Lxr-dev] Swish-E
>
>
> > On Wed, 2001-11-21 at 23:04, Arne Georg Gleditsch wrote:
> > > Patch follows.  Note though, that the way I've implemented this seems
> > > to require Swish-E newer than the currently stable 2.0.5.  This might
> > > or might not be a problem.  Anyway, have a look at it.
> >
> > I tried the non-development version during the weekend and found the
> > indexer to be extremely slow. It used a lot of memory, and made my
> > machine crawl under heavy swappping. It actually took so long that I
> > never got around to start writing any code for using Swish-E, so it's
> > great that you got around to do it.
> >
> >                                            Per Kristian
> >
> >
> > _______________________________________________
> > Lxr-developer mailing list
> > Lxr...@li...
> > https://lists.sourceforge.net/lists/listinfo/lxr-developer
>

[Lxr-dev] [ lxr-Feature Requests-489929 ] function index (speedbar)

[<no...@so...>]

Feature Requests item #489929, was opened at 2001-12-06 09:53
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=489929&group_id=27350

Category: Browsing Interface
Group: None
Status: Open
Priority: 5
Submitted By: David Oleszkiewicz (poppinfresh)
Assigned to: Nobody/Anonymous (nobody)
Summary: function index (speedbar)

Initial Comment:
I use the speedbar in xemacs alot to help be jump 
straight down to a function in a source file.  Well i 
actually map it to the mouse...but it would be nice 
to have a speebar (html frame) on the the left or 
right hand side of the source file which has links to 
all of the functions declared in a source file.  
presumably this information is already parsed in a 
manner that you know it is a function def.  this 
could be ignored in header files or contain the index 
of typedefs and #defines or something.  


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390120&aid=489929&group_id=27350

[Lxr-dev] [ lxr-Bugs-489866 ] --allversions doesn't work

[<no...@so...>]

Bugs item #489866, was opened at 2001-12-06 06:47
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=489866&group_id=27350

Category: genxref
Group: current cvs
Status: Open
Resolution: None
Priority: 8
Submitted By: Malcolm Box (mbox)
Assigned to: Peder O. Klingenberg (pok)
Summary: --allversions doesn't work

Initial Comment:
This is a tracking bug for the the CVS.pm changes
needed to make --allversions work properly across CVS
and Files.

Currently one or the other works - we need both to work
with the same semantics before 1.00

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=489866&group_id=27350