lxr-developer — Don't use, active developers only

Re: [Lxr-dev] Re: /bin/true as a zombie

[<pe...@kl...>]

Jan-Benedict Glaw <jb...@lu...> writes:

> [...]
> ### /lib/wait.c  
> --- /lib/wait.c head 1.2
> /lib/wait.c was already referenced
> ### /lib/write.c  
> --- /lib/write.c head 1.2
> /lib/write.c was already referenced

Did you empty the database before running genxref?  Otherwise you may
have stuff lying around in the tables that makes genxref think you
already indexed these files.

> ### /mm/Makefile  
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
> co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low

Hm.  It gets as far as trying to check out the file, but seems to
somehow get confused over which version to check out.  Trying six
times would indicate that it's trying to check out six different
versions.  Or something.

> Which is also triggered with the older CVS version of gnexref you mentioned
> above.

Yep.  It's a bug in the database interface libraries.

> I've also tried the old copy of ping's genxref version. It triggers the
> "branch number 0 too low" bug also

Yes, it should, as that message is coming from CVS.pm.

> If it could help you, I'd package my complete LXR installation (
> functional and non-functional as well) and send it to you...

Yes, thank you, that would be good.  Especially if you include the CVS
tree you're trying to index (it's a good thing it's linux kernels, so
I don't have to sign anything :) ).  I don't need the ping version,
only the sourceforge version.

I can't promise you that I'll fix this quickly (having to work for a
living as well :) ), but it's certainly easier for me to debug
something I can edit and hack around with myself than having to debug
by mail.

...Peder...
-- 
Cogito ergo panta rei.

Re: [Lxr-dev] Re: /bin/true as a zombie

[Jan-Benedict G. <jb...@lu...>]

On Tue, Jul 31, 2001 at 10:57:30AM +0200, Peder O. Klingenberg wrote:
> Jan-Benedict Glaw <jb...@lu...> writes:
> and less obvious, but also less broken way (for CVS trees).  Could you
> try --allversions with that version of genxref?  It's available from
> <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/~checkout~/lxr/lxr/genxref?rev=1.18&content-type=text/plain>,
> and I think thats the same version as in the old ping CVS tree.

- I've reverted the change to ./lib/LXR/CVS.pm
- Checked out the above version og genxref
- Result:
	- This version also *only* tries to index head
	- It seems to handle some things wrong:

[...]
### /lib/wait.c  
--- /lib/wait.c head 1.2
/lib/wait.c was already referenced
### /lib/write.c  
--- /lib/write.c head 1.2
/lib/write.c was already referenced
### /mm/  
### /mm/RCS/  
### /mm/Makefile  
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
co: /home/data/CVSROOT/linux//mm/Makefile,v: branch number 0 too low
### /mm/memory.c  
--- /mm/memory.c head 1.4
/mm/memory.c was already referenced
### /mm/page.s  
[...]

> > disconnect(DBI::db=HASH(0x83a7a30)) invalidates 1 active statement. Either destroy statement handles or call finish on them before disconnecting. at lib//LXR/Index/Postgres.pm line 272.
> 
> This is an unrelated, but nevertheless irritating bug.

Which is also triggered with the older CVS version of gnexref you mentioned
above.

I've also tried the old copy of ping's genxref version. It triggers the
"branch number 0 too low" bug also

> > Yes. Another copy of LXR (that one from the CVS tree on cvs.ping.uio.no)
> > running on the same source tree(but from another directory with its
> > own config etc.) behaves just fine with it:-)
> 
> weird.

If it could help you, I'd package my complete LXR installation (
functional and non-functional as well) and send it to you...

> > It doesn't exit ungracefully. Just like above (with cancelling one
> > active statement by closing the DB handle too early). My CVS
tree is also not that large (~1.5MB) uncompressed...

> And I don't think genxref explains why you're having trouble
> browsing, so that must be a separate issue.

I'm unsure. Those oloder versions show up this "branch 0 too low"
thing I could not associate to any usage of CVS I ever had when
accessing it by hand...

MFG, JBG

Re: [Lxr-dev] Re: /bin/true as a zombie

[<pe...@kl...>]

Jan-Benedict Glaw <jb...@lu...> writes:

> Doing "allversions" results in indexing head:

Ugh.  There was a change in genxref recently that tidied up a great
deal, but unfortunately left 'allversions' utterly broken, despite the
log comment that it made allversions work.  That "fix" assumed that
versions are uniform throughout the tree.  That is usually not the
case in a CVS tree, of course.  I don't know what, if anything, was
wrong with allversions in the old code for plaintext files.

The previous version (1.18) of genxref did this in a more convoluted
and less obvious, but also less broken way (for CVS trees).  Could you
try --allversions with that version of genxref?  It's available from
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/~checkout~/lxr/lxr/genxref?rev=1.18&content-type=text/plain>,
and I think thats the same version as in the old ping CVS tree.

> disconnect(DBI::db=HASH(0x83a7a30)) invalidates 1 active statement. Either destroy statement handles or call finish on them before disconnecting. at lib//LXR/Index/Postgres.pm line 272.

This is an unrelated, but nevertheless irritating bug.

> Yes. Another copy of LXR (that one from the CVS tree on cvs.ping.uio.no)
> running on the same source tree(but from another directory with its
> own config etc.) behaves just fine with it:-)

weird.

> It doesn't exit ungracefully. Just like above (with cancelling one
> active statement by closing the DB handle too early).

Ok.  Revert that change, then.  If the old version of genxref works
with allversions, could you try it with just your specific version?
If that does not work, we'll have to do some more debugging of
CVS.pm.

And I don't think genxref explains why you're having trouble
browsing, so that must be a separate issue.

> > Cogito ergo panta rei.
> 
> Why do you mux latin and greek(sp?) ? :-)

Why not? :) With a bit of goodwill, it reads "I think, therefore
everything is in flux".  In norwegian, "everything is in flux" is also
an expression of confusion, which is my usual state of mind when I'm
thinking.  Besides, it's a bit confusing mixing latin and greek,
Descartes an Heraklit.

I must admit it's not mine, it's originally reputed to be the answer
of a poor, confused student on a philosophy exam.  But I think it's
funny none the less. :)

...Peder...
-- 
Cogito ergo panta rei.

[Lxr-dev] Re: /bin/true as a zombie

[Jan-Benedict G. <jb...@lu...>]

On Tue, Jul 31, 2001 at 09:34:12AM +0200, Peder O. Klingenberg wrote:
> Jan-Benedict Glaw <jb...@lu...> writes:
> > The current CVS version cannot ./genxref my sources. It only prints
> > two lines:
> > www-data@mirror:/home/data/lxr-cvs$ ./genxref --url="http://mirror.microdata-pos.de/lxr-cvs" --version="jbglaw_0_13"
> > *** / jbglaw_0_13 
> > ### / jbglaw_0_13 
> > 
> > "jbglaw_0_13" is a tag.
> 
> I always index all versions.  Could you try that as well, and see if
> it makes a difference?

The CVS repository contains very early Linux kernels. Tags are
"v_0_01", "v_0_10", "v_0_11", "v_0_12", "jbglaw_0_13".

Doing "allversions" results in indexing head:

www-data@mirror:/home/data/lxr-cvs$ ./genxref --url="http://mirror.microdata-pos.de/lxr-cvs" --allversions
[...]
### /Makefile head 
disconnect(DBI::db=HASH(0x83a7a30)) invalidates 1 active statement. Either destroy statement handles or call finish on them before disconnecting. at lib//LXR/Index/Postgres.pm line 272.
www-data@mirror:/home/data/lxr-cvs$ 

This was the 2nd run over the tree. It doesn't see the other tags:-(


> > Tryin' to get files (read: HTML made-up sources) from it fails, too:
> > 
> >         The directory /home/data/CVSROOT/linux// does not exist.
> >           ** Warning: Unable to open /home/data/CVSROOT/linux//
> > 
> > Of course, everything *is* in place:-) 
> 
> And readable for the webserver?

Yes. Another copy of LXR (that one from the CVS tree on cvs.ping.uio.no)
running on the same source tree(but from another directory with its
own config etc.) behaves just fine with it:-)

> > Do you know what I could do wrong?
> 
> No, not really.  It's obvious that genxref never recurses down
> directories like it's supposed to.  Which means that CVS.pm->getdir
> somehow fails.  I haven't touched that part of CVS.pm, so I don't
> think _I've_ broken it.  That doesn't mean it's not broken :)
> 
> Could you try to replace line 220 in CVS.pm with this:
> 
> 	opendir($DIRH, $real) || die "opendir: $!";
> 
> And see if that changes anything when you run genxref?  If genxref
> exits with an errormessage "opendir: ..." could you post it?

It doesn't exit ungracefully. Just like above (with cancelling one
active statement by closing the DB handle too early).

> Cogito ergo panta rei.

Why do you mux latin and greek(sp?) ? :-)

MfG, JBG

[Lxr-dev] Re: /bin/true as a zombie

[<pe...@kl...>]

Jan-Benedict Glaw <jb...@lu...> writes:

> Well, I've checked the current CVS version. It fails.

Strange.

> Configuration uses Postgres in conjunction with CVS (as I mentioned
> before). 

This is what I use as well.  No problems here (well, not this kind of
problems anyway).

> The current CVS version cannot ./genxref my sources. It only prints
> two lines:
> www-data@mirror:/home/data/lxr-cvs$ ./genxref --url="http://mirror.microdata-pos.de/lxr-cvs" --version="jbglaw_0_13"
> *** / jbglaw_0_13 
> ### / jbglaw_0_13 
> 
> "jbglaw_0_13" is a tag.

I always index all versions.  Could you try that as well, and see if
it makes a difference?

> Tryin' to get files (read: HTML made-up sources) from it fails, too:
> 
>         The directory /home/data/CVSROOT/linux// does not exist.
>           ** Warning: Unable to open /home/data/CVSROOT/linux//
> 
> Of course, everything *is* in place:-) 

And readable for the webserver?

> Do you know what I could do wrong?

No, not really.  It's obvious that genxref never recurses down
directories like it's supposed to.  Which means that CVS.pm->getdir
somehow fails.  I haven't touched that part of CVS.pm, so I don't
think _I've_ broken it.  That doesn't mean it's not broken :)

Could you try to replace line 220 in CVS.pm with this:

	opendir($DIRH, $real) || die "opendir: $!";

And see if that changes anything when you run genxref?  If genxref
exits with an errormessage "opendir: ..." could you post it?

Thanks.

...Peder...
-- 
Cogito ergo panta rei.

[Lxr-dev] Re: /bin/true as a zombie

[Jan-Benedict G. <jb...@lu...>]

On Mon, Jul 30, 2001 at 01:34:03PM +0200, Peder O. Klingenberg wrote:
> > Is there any solutiion to avoid the zombie or to fetch the SIGCHLD and
> > to wait() for it? Personally, I don't have any clue about Perl so
> > I cannot fix it myself:-(
> 
> Hopefully, this is fixed in the current CVS tree.
> 
> By the way, you are aware of the move to sourceforge?  There's some
> new mailing lists and a new CVS repository.  See
> http://sourceforge.net/projects/lxr/.

Well, I've checked the current CVS version. It fails.

Configuration uses Postgres in conjunction with CVS (as I mentioned
before). The current CVS version cannot ./genxref my sources. It
only prints two lines:
www-data@mirror:/home/data/lxr-cvs$ ./genxref --url="http://mirror.microdata-pos.de/lxr-cvs" --version="jbglaw_0_13"
*** / jbglaw_0_13 
### / jbglaw_0_13 

"jbglaw_0_13" is a tag.

Tryin' to get files (read: HTML made-up sources) from it fails, too:

        The directory /home/data/CVSROOT/linux// does not exist.
          ** Warning: Unable to open /home/data/CVSROOT/linux//

Of course, everything *is* in place:-) strace'ing it shows that LXR
stat()s all files in the given directory, but doesn't actually do
something.

Do you know what I could do wrong?

MfG, JBG

[Lxr-dev] Some patches for review (CVS tree indexing working)

[<pe...@kl...>]

Finally, I have a version of LXR working that indexes our CVS tree
directly.

Here's the significant patches against the current CVS tree on
sourceforge.  I'd be very happy if someone took a look and told me if
there was something fundamentally wrong with this (I'd also appreciate
being told what a genius I am, naturally ;) ).

If I don't hear anything, I will assume the changes don't bother
anyone, and check them in in a day or two.

The bulk of the changes are in CVS.pm, which now doesn't segfault
anymore when faced with binary files.  It now largely uses external
commands for getting files out of CVS.  Unfortunately, it does this in
a way that is probably exploitable.  While this is not a big issue on
our intranet, you'd probably not want to run this on a public server
yet.  

My next task will be to fix this.  I just wanted to get these patches
out the door, as they've taken far too long already.

See also comments in between the parts of the patch.

===================================================================

These patches to the datamodel may be just me being dense, but because
a CVS file contains a lot of versions, there is no single mapping
file->version.  genxref complained loudly without this change.  Also,
the usage table wasn't dropped along with the others in postgres.

Index: initdb
===================================================================
RCS file: /cvsroot/lxr/lxr/initdb,v
retrieving revision 1.10
diff -u -r1.10 initdb
--- initdb	1999/09/17 09:37:37	1.10
+++ initdb	2001/07/24 11:16:34
@@ -4,6 +4,7 @@
 drop table symbols;
 drop table indexes;
 drop table releases;
+drop table usage;
 drop table status;
 
 create sequence filenum cache 50;
@@ -37,7 +38,7 @@
 create table releases 
 	(fileid		int		references files,
 	release		varchar,
-	primary key	(fileid)
+	primary key	(fileid,release)
 );
 
 create table usage
Index: initdb-mysql
===================================================================
RCS file: /cvsroot/lxr/lxr/initdb-mysql,v
retrieving revision 1.4
diff -u -r1.4 initdb-mysql
--- initdb-mysql	2001/05/23 05:33:12	1.4
+++ initdb-mysql	2001/07/24 11:16:34
@@ -34,7 +34,7 @@
 create table releases 
         (fileid         int not null references files,
         release         char(255) binary not null,
-        primary key     (fileid)
+        primary key     (fileid,release)
 );
 
 create table useage
Index: initdb-postgres
===================================================================
RCS file: /cvsroot/lxr/lxr/initdb-postgres,v
retrieving revision 1.1
diff -u -r1.1 initdb-postgres
--- initdb-postgres	1999/12/25 21:58:27	1.1
+++ initdb-postgres	2001/07/24 11:16:34
@@ -4,6 +4,7 @@
 drop table symbols;
 drop table indexes;
 drop table releases;
+drop table usage;
 drop table status;
 
 create sequence filenum cache 50;
@@ -37,7 +38,7 @@
 create table releases 
 	(fileid		int		references files,
 	release		varchar,
-	primary key	(fileid)
+	primary key	(fileid,release)
 );
 
 create table usage
===================================================================

The comment says it all, really.  I just wanted to illustrate a new
way of specifying the range.

Index: lxr.conf.template
===================================================================
RCS file: /cvsroot/lxr/lxr/lxr.conf.template,v
retrieving revision 1.5
diff -u -r1.5 lxr.conf.template
--- lxr.conf.template	1999/06/16 09:17:48	1.5
+++ lxr.conf.template	2001/07/24 11:16:34
@@ -22,6 +22,15 @@
 	 # Define typed variable "v", read valueset from file.
 	 'v' => {'name'    => 'Version',
 		 'range'   => [ readfile('src/cvsversions') ], 
+
+                 # If files within a tree can have different versions,
+		 # e.g in a CVS tree, 'range' can be specified as a
+		 # function to call for each file:
+		 #'range'   => sub { return 
+		 #			($files->allreleases($LXR::Common::pathname),
+		 #			 $files->allrevisions($LXR::Common::pathname))
+		 #			}, # deferred function call.
+
 		 'default' => '1.0.6'},
 	 
 	 # Define typed variable "a".  First value is default.
===================================================================

And this is what implements this new way of specifying the range.

Index: lib/LXR/Config.pm
===================================================================
RCS file: /cvsroot/lxr/lxr/lib/LXR/Config.pm,v
retrieving revision 1.23
diff -u -r1.23 Config.pm
--- lib/LXR/Config.pm	2000/09/04 19:26:28	1.23
+++ lib/LXR/Config.pm	2001/07/24 11:16:36
@@ -117,6 +117,10 @@
 sub varrange {
     my ($self, $var) = @_;
 
+	if (ref($self->{variables}{$var}{range}) eq "CODE") {
+		return &{$self->{variables}{$var}{range}};
+	}
+
     return @{$self->{variables}{$var}{range} || []};
 }

===================================================================

The shebang parsing didn't work previously.  Now it does.

Index: lib/LXR/Lang.pm
===================================================================
RCS file: /cvsroot/lxr/lxr/lib/LXR/Lang.pm,v
retrieving revision 1.19
diff -u -r1.19 Lang.pm
--- lib/LXR/Lang.pm	1999/12/25 21:58:27	1.19
+++ lib/LXR/Lang.pm	2001/07/24 11:16:38
@@ -30,9 +30,9 @@
 		$lang = new LXR::Lang::Perl($pathname, $release);
 	}
 	else {
-		my ($shebang);
+		$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;
 
-		$shebang = $files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;
+		my $shebang = $1;
 
 		if ($shebang =~ /perl/) {
 			require LXR::Lang::Perl;
===================================================================

Quite a lot of changes here.  Some insignificant bugfixes for
undefined values and stuff, and then the big change of having parsecvs
only parse the header of the cvs file, because it will get confused by
binary content in the controlled file.  All places that depended on
the content of the file being available in the %cvs hash have been
changed to use the proper RCS/CVS commands instead (unfortunately not
in a safe way - yet).

Index: lib/LXR/Files/CVS.pm
===================================================================
RCS file: /cvsroot/lxr/lxr/lib/LXR/Files/CVS.pm,v
retrieving revision 1.11
diff -u -r1.11 CVS.pm
--- lib/LXR/Files/CVS.pm	1999/11/24 14:48:53	1.11
+++ lib/LXR/Files/CVS.pm	2001/07/24 11:16:38
@@ -29,6 +29,9 @@
 	if ($release =~ /rev_([\d\.]+)/) {
 		return $1;
 	}
+	elsif ($release =~ /([\d\.]+)/) {
+		return $1;
+	}
 	else {
 		$self->parsecvs($filename);
 		return $cvs{'header'}{'symbols'}{$release};
@@ -47,8 +50,10 @@
 	return undef unless defined($rev);
 
 	my @t = reverse(split(/\./, $cvs{'branch'}{$rev}{'date'}));
-	$t[4]--;
 
+	return undef unless @t;
+
+	$t[4]--;
 	return timegm(@t);
 }
 
@@ -60,38 +65,10 @@
 
 sub getfile {
 	my ($self, $filename, $release) = @_;
-
-	$self->parsecvs($filename);
-
-	my $rev = $self->filerev($filename, $release);
-	return undef unless defined($rev);
-
-	my $hrev = $cvs{'header'}{'head'};
-	my @head = $cvs{'history'}{$hrev}{'text'} =~ /([^\n]*\n)/gs;
-
-	while ($hrev ne $rev && $cvs{'branch'}{$hrev}{'branches'} ne $rev) {
-		$hrev = $cvs{'branch'}{$hrev}{'next'};
-		my @diff = $cvs{'history'}{$hrev}{'text'} =~ /([^\n]*\n)/gs;
- 		my $off = 0;
-
-		while (@diff) {
-			my $dir = shift(@diff);
 
-			if ($dir =~ /^a(\d+)\s+(\d+)/) {
-				splice(@head, $1-$off, 0, splice(@diff, 0, $2));
-				$off -= $2;
-			}
-			elsif ($dir =~ /^d(\d+)\s+(\d+)/) {
-				splice(@head, $1-$off-1, $2);
-				$off += $2;
-			}
-			else {
-				warning("Oops! Out of sync!");
-			}
-		}
-	}
-
-	return join('', @head);
+	my $fileh = $self->getfilehandle($filename, $release);
+	return undef unless $fileh;
+	return join('', $fileh->getlines);
 }
 
 sub getannotations {
@@ -105,22 +82,23 @@
 	my $hrev = $cvs{'header'}{'head'};
 	my $lrev;
 	my @anno;
-	my @head = $cvs{'history'}{$hrev}{'text'} =~ /\n()/gs;
+	my $headfh = $self->getfilehandle($filename, $release);
+	my @head = $headfh->getlines;
 
 	while (1) {
 		if ($rev eq $hrev) {
 			@head = 0..$#head;
 		}
-
+		
 		$lrev = $hrev;
 		$hrev = $cvs{'branch'}{$hrev}{'next'} || last;
-
-		my @diff = $cvs{'history'}{$hrev}{'text'} =~ /([^\n]*\n)/gs;
- 		my $off = 0;
-
+		
+		my @diff = $self->getdiff($filename, $lrev, $hrev);
+		my $off = 0;
+		
 		while (@diff) {
 			my $dir = shift(@diff);
-
+			
 			if ($dir =~ /^a(\d+)\s+(\d+)/) {
 				splice(@diff, 0, $2);
 				splice(@head, $1-$off, 0, ('') x $2);
@@ -130,7 +108,7 @@
 				map {
 					$anno[$_] = $lrev if $_ ne '';
 				} splice(@head, $1-$off-1, $2);
-
+				
 				$off += $2;
 			}
 			else {
@@ -159,22 +137,33 @@
 	my ($self, $filename, $release) = @_;
 	my ($fileh);
 
-#	$fileh = new FileHandle("co -q -pv$release ".
-#							$self->toreal($filename, $release).
-#							" |"); # FIXME: Exploitable?
-
-	my $buffer = $self->getfile($filename, $release);
-
-	&LXR::Common::fflush;
-	my ($readh, $writeh) = FileHandle::pipe;
-	unless (fork) {
-		$writeh->autoflush(1);
-		$writeh->print($buffer);
-		exec("/bin/true");		# Exit without cleanup.
-		exit;
-	}
+	$self->parsecvs($filename);
+
+	my $rev = $self->filerev($filename, $release);
+	return undef unless defined($rev);
+
+	$fileh = new FileHandle("co -q -p$rev ".
+							$self->toreal($filename, $release).
+							" |"); # FIXME: Exploitable?
+	return $fileh;
+}
+
+sub getdiff {
+	my ($self, $filename, $release1, $release2) = @_;
+	my ($fileh);
+
+	$self->parsecvs($filename);
+
+	my $rev1 = $self->filerev($filename, $release1);
+	return undef unless defined($rev1);
+
+	my $rev2 = $self->filerev($filename, $release2);
+	return undef unless defined($rev2);
 
-	return $readh;
+	$fileh = new FileHandle("rcsdiff -q -a -n -r$rev1 -r$rev2 ".
+							$self->toreal($filename, $release1).
+							" |"); # FIXME: Exploitable?
+	return $fileh->getlines;
 }
 
 sub tmpfile {
@@ -297,15 +286,35 @@
 	return sort(keys(%{$cvs{'header'}{'symbols'}}));
 }
 
+sub allrevisions {
+	my ($self, $filename) = @_;
+
+	$self->parsecvs($filename);
+
+	return sort(keys(%{$cvs{'branch'}}));
+}
+
 sub parsecvs {
+	# Actually, these days it just parses the header.
+	# RCS tools are much better at parsing RCS files.
+	# -pok
 	my ($self, $filename) = @_;
 
 	return if $cache_filename eq $filename;
 	$cache_filename = $filename;
+
+	my $file = '';
+	open (CVS, $self->toreal($filename, undef));
+	while (<CVS>) {
+		if (/^text\s*$/) {
+			# stop reading when we hit the text.
+			last;
+		}
+		$file .= $_;
+	}
+	close (CVS);
 
-	open(CVS, $self->toreal($filename, undef));
-	my @cvs = join('', <CVS>) =~ /((?:(?:[^\n@]+|@[^@]*@)\n?)+)/gs;
-	close(CVS);
+	my @cvs = $file =~ /((?:(?:[^\n@]+|@[^@]*@)\n?)+)/gs;
 
 	$cvs{'header'} = { map { s/@@/@/gs;
 							 /^@/s && substr($_, 1, -1) || $_ }
@@ -313,7 +322,7 @@
 
 	$cvs{'header'}{'symbols'}
 	= { $cvs{'header'}{'symbols'} =~ /(\S+?):(\S+)/g };
-
+	
 	my ($orel, $nrel, $rev);
 	while (($orel, $rev) = each %{$cvs{'header'}{'symbols'}}) {
 		$nrel = $config->cvsversion($orel);
@@ -337,13 +346,6 @@
 	$cvs{'desc'} = shift(@cvs) =~ /\s*desc\s+((?:[^\n@]+|@[^@]*@)*)\n/s;
 	$cvs{'desc'} =~ s/^@|@($|@)/$1/gs;
 
-	while (@cvs) {
-		my ($r, $v) = shift(@cvs) =~ /\s*(\S+)\s*(.*)/s;
-		$cvs{'history'}{$r} = { map { s/@@/@/gs; 
-									  /^@/s && substr($_, 1, -1) || $_ }
-								$v =~ /(\w+)\s*((?:[^\n@]+|@[^@]*@)*)\n/gs };
-	}
 }
-
 
 1;
===================================================================

Also, some small changes to the Postgres interface.  I dislike
hardcoded numbers that occur more that one place, so I put the number
of inserts before a commit into a variable.  I also fixed a bug
resulting from a wrong assumption on the return value on an execute of
a select statement.

Index: lib/LXR/Index/Postgres.pm
===================================================================
RCS file: /cvsroot/lxr/lxr/lib/LXR/Index/Postgres.pm,v
retrieving revision 1.4
diff -u -r1.4 Postgres.pm
--- lib/LXR/Index/Postgres.pm	2000/10/31 12:52:12	1.4
+++ lib/LXR/Index/Postgres.pm	2001/07/24 11:16:41
@@ -9,7 +9,7 @@
 use strict;
 use DBI;
 
-use vars qw($dbh $transactions %files %symcache 
+use vars qw($dbh $transactions %files %symcache $commitlimit
 			$files_select $filenum_nextval $files_insert
 			$symbols_byname $symbols_byid $symnum_nextval
 			$symbols_remove $symbols_insert $indexes_select $indexes_insert
@@ -26,7 +26,8 @@
 
 	$$dbh{'AutoCommit'} = 0;
 #	$dbh->trace(1);
-	
+
+	$commitlimit = 100;
 	$transactions = 0;
 	%files = ();
 	%symcache = ();
@@ -96,7 +97,7 @@
 							 $line,
 							 $type,
 							 $relsym ? $self->symid($relsym) : undef);
-	unless (++$transactions % 500) {
+	unless (++$transactions % $commitlimit) {
 		$dbh->commit();
 	}
 }
@@ -108,7 +109,7 @@
 						   $line,
 						   $self->symid($symname));
 
-	unless (++$transactions % 500) {
+	unless (++$transactions % $commitlimit) {
 		$dbh->commit();
 	}
 }
@@ -178,12 +179,16 @@
 # Indicate that this filerevision is part of this release
 sub release {
 	my ($self, $fileid, $release) = @_;
+
+
+	$releases_select->execute($fileid+0, $release);
+	my $firstrow = $releases_select->fetchrow_array();
+
 
-	my $rows = $releases_select->execute($fileid+0, $release);
-	$releases_select->finish();
+#	$releases_select->finish();
 
-	unless ($rows > 0) {
-		$releases_insert->execute($fileid, $release);
+	unless ($firstrow) {
+		$releases_insert->execute($fileid+0, $release);
 	}
 }
 
@@ -239,7 +244,7 @@
 	my ($self, $fileid) = @_;
 
 	$status_insert->execute($fileid+0, $fileid+0);
-	return $status_update->execute(1, $fileid, 0) > 0;
+	return $status_update->execute(1, $fileid+0, 0) > 0;
 }
 
 sub toreference {


...Peder...
-- 
Cogito ergo panta rei.

[Lxr-dev] Hello to all...also some news & Qs

[Nadeem H. <nh...@na...>]

Hi all,

This message is long so here is the summary:

1. Changes were necessary in 0.3 to make it run under mod_perl.
2. Added the image preview/info functionality to LXR. It supports
   PNG, GIF, JPG, BMP, XPM file formats.
3. Added some more niceties.
4. Found C++ handling to be a bit buggy.

I just joined the list and wanted to say hello. I am using LXR to
run the http://lxr.kde.org/ site, the KDE Cross Reference. I started
out with 0.3 code and got it working under CGI after some usual
configuration and set up gotchas :).

When I decided to move to mod_perl, I found out that the code was
not written with mod_perl in mind. so I had to make a few changes
to fix that. That mainly meant initializing the variables before
they are used.

KDE has a lot of image files and LXR was not image friendly. It
just returned the image data which got displayed as junk. So I
went ahead and wrote two scripts to add image preview capability.
Check it out here:

http://kdelxr.nadmm.com/imageinfo/kdegraphics/kiconedit/pics/hi48-app-kiconedit.png

I also made some more minor enhancements like using appropriate
icons for image & README files in the index listing.

I want to contribute these back to LXR project if there is an 
interest in it. How can I do that? Just post a patch here?
I gues I may have to apply the changes first to the latest CVS
version and post it here.

I also found that C++ cross reference is a bit buggy and returns
too many false hits. I want to know if there is any improvement
in it after the 0.3 version. I am depending on it. Anyway, I'll
try the CVS code soon and try to forward port my changes to it.

Thanks for the patience.

Cheers,
--
Nadeem Hasan
nh...@na...
http://www.nadmm.com/

[Lxr-dev] [ lxr-Patches-438564 ] Bug fix for bug 438492

[<no...@so...>]

Patches item #438564, was opened at 2001-07-04 08:43
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390119&aid=438564&group_id=27350

Category: Other
Group: Bugfix
Status: Open
Resolution: None
Priority: 5
Submitted By: Jens-Uwe Mager (jum)
Assigned to: Nobody/Anonymous (nobody)
Summary: Bug fix for bug 438492

Initial Comment:
This clears the array that is used to collect the 
glimpse results on each request to make sure no 
old search results from previous invocations under  
Apache/mod_perl are returned.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390119&aid=438564&group_id=27350

[Lxr-developer] [ lxr-Bugs-438492 ] Glimpse returns the wrong results

[<no...@so...>]

Bugs item #438492, was opened at 2001-07-04 03:46
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=438492&group_id=27350

Category: Browsing
Group: current cvs
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Glimpse returns the wrong results

Initial Comment:
Having applied the glimpse patch to get freetext
indexing working things work OK but occasionally the
wrong results are returned for a search.  Specifically,
the results from a previous search are returned, and
multiple reloads of the page are required to cause it
to show the right results.

I assume this is something to do with Apache and
mod_perl not destroying the script at the end of each
invocation.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=438492&group_id=27350

[Lxr-developer] Re: Modified LXR for TclCL

[Wesley M. <we...@bm...>]

I've posted a tar.gz of my modified source (merged with the official LXR
0.3 source) to http://www.openmash.org/lxr/. In addition to supporting
identifiers for TclCL, it also recognizes a special variable named
'Component' and treats them differently when generating the HTML
headers. I wanted to specify components seperately, and wrote some stuff
to do so, but ultimately it was easier to just hack in the $string eq
'Component' and handle it differently.

- Wes

Per Kristian Gjermshus wrote:
> 
> * Wesley Miaw
> | I wanted to drop you a line and let you know that I modified LXR so
> | that it will work on source trees containing both C/C++ and TclCL. It
> | creates an additional database for TclCL identifiers (which break down
> | into classes, global variables, and functions) and cross-references
> | subclasses of the C++ class TclClass (but not functions of such
> | classes).
> |
> | I can send you my modified source code if you like. I think we will
> | also end up posting it somewhere on the site where we use it:
> | http://www.openmash.org/.
> 
> That would certainly be interesting. I am cc'ing the developers list
> as well. Someone might be interested in integrating your work with the
> development branch of LXR.
> 
>                                       Per Kristian

[Lxr-developer] Re: Modified LXR for TclCL

[Per K. G. <pk...@ne...>]

* Wesley Miaw
| I wanted to drop you a line and let you know that I modified LXR so
| that it will work on source trees containing both C/C++ and TclCL. It
| creates an additional database for TclCL identifiers (which break down
| into classes, global variables, and functions) and cross-references
| subclasses of the C++ class TclClass (but not functions of such
| classes).
| 
| I can send you my modified source code if you like. I think we will
| also end up posting it somewhere on the site where we use it:
| http://www.openmash.org/.

That would certainly be interesting. I am cc'ing the developers list
as well. Someone might be interested in integrating your work with the
development branch of LXR.

                                      Per Kristian

Re: [Lxr-developer] Code documentation

[Per K. G. <pk...@ne...>]

* Neil Salter
| The guys here: http://unstable.elemental.com/mozilla/ had the idea of cross
| linking between doxygen and LXR. I think this is a good idea. Currently I'm
| using doxygen to document a (fairly large) source tree, and I plan to use
| LXR on the same tree. I'd say it'd be pretty useful to be able to jump from,
| say, a classname in LXR to the doxygen page for the class - and vice versa.
| It could all be done manually by the user just now, but it might be worth
| thinking about as potential future feature for incorporation into the
| software.

Supporting documentation systems and bug tracking systems is something
we have been thinking about. My personal opinion is to not tie LXR to
any particular system, but provide general hooks that would make it
straightforward to support more than one system. 

Besides from that I think it would be smarter to concentrate on
getting the basic functionality in place before starting to add new
features. But, this is an open source project, people are free to work
on whatever they want.

                                             Per Kristian

Re: [Lxr-developer] Doco template

[Per K. G. <pk...@ne...>]

* Zubin Sethna
| We can though manually 'add' the missing base class documentation. Their
| should be no need to add 'empty' functions since POD (AFAIK) is pretty brain
| dead and doesn't actually parse the perl code in anyway except to look for
| POD commands (between lines beginning with '=pod' and '=cut').

POD is nice and useful. The problem is of course that it does not
write the documentation for you. :-) The perl module of LXR itself
should understand POD and do something interesting with it.

                                                   Per Kristian

RE: [Lxr-developer] Doco template

[Sethna, Z. (R. Brisbane) <ZS...@rs...>]

From what I have read about POD so far, it doesn't have any support for OO
constructs, unlike doxygen but then doxygen doesn't support perl yet :-{{

We can though manually 'add' the missing base class documentation. Their
should be no need to add 'empty' functions since POD (AFAIK) is pretty brain
dead and doesn't actually parse the perl code in anyway except to look for
POD commands (between lines beginning with '=pod' and '=cut').

Zubin


-----Original Message-----
From: Malcolm Box [mailto:ma...@br...]
Sent: Thursday, 21 June 2001 1:02 AM
To: lxr...@li...
Subject: Re: [Lxr-developer] Doco template


Hi Zubin,

"Sethna, Zubin (RSA, Brisbane)" wrote:
> I want to propose that we us the following file header and function
template
> (in perl's POD format) to document the LXR source code. Tried to keep it
> simple as possible.
<snip>

This looks good and simple which is always an advantage.  Being an OO
system there are lots of classes in the lxr - do you have a suggestion
for documenting them?  I don't know what the Perl Way is for this.

For example, what I'd really like to do (actually see done :-) is
documentation for each of the "base" classes Index, File & Lang to
define what the methods are and what the usage/expected return values
are.  At the moment it's only possible to deduce this by looking at one
of the derived classes, which risks confusing the interface and the
implementation.  I was considering adding empty functions to the base
classes as placeholders for derived code & possibly as a place for the
documentation.

Cheers,

Malcolm

_______________________________________________
Lxr-developer mailing list
Lxr...@li...
http://lists.sourceforge.net/lists/listinfo/lxr-developer

Re: [Lxr-developer] Weird MySQL bug

[Malcolm B. <ma...@br...>]

"Peder O. Klingenberg" wrote:
> 
> Malcolm Box <ma...@br...> writes:
> 
> > $dbh, which is the database handle in question, is a package
> > variable and therefore I think it should be scoped until the package
> > is removed, ie when the END block deletes things properly.
> 
> I would tend to agree with you, FWIW.  I don't use mysql myself, so I
> can't experiment, but have you tried

Hmm, I wonder if similar things show up with pgsql and if not, what the
code differences are.
 
>  * Setting $dbh->trace(n) with n>0 right after you connect?
> 
>  * Explicitly qualifying $dbh with the package, ie
>    s/$dbh/$LXR::Index::Mysql::dbh/ ?
> 
>  * A different version of perl?

I'm running 5.6.0 which is one down from the latest stable so I may try
the update.  I'll try the other things you suggest and see what turns
up.  This one is really baffling me at the moment!

> Not that I would know what was wrong given results of those tests, but
> maybe they can help shed light on the problem?

Hell, I'll try anything at the moment - shotgun debugging it is!

Malcolm

Re: [Lxr-developer] Doco template

[Malcolm B. <ma...@br...>]

Hi Zubin,

"Sethna, Zubin (RSA, Brisbane)" wrote:
> I want to propose that we us the following file header and function template
> (in perl's POD format) to document the LXR source code. Tried to keep it
> simple as possible.
<snip>

This looks good and simple which is always an advantage.  Being an OO
system there are lots of classes in the lxr - do you have a suggestion
for documenting them?  I don't know what the Perl Way is for this.

For example, what I'd really like to do (actually see done :-) is
documentation for each of the "base" classes Index, File & Lang to
define what the methods are and what the usage/expected return values
are.  At the moment it's only possible to deduce this by looking at one
of the derived classes, which risks confusing the interface and the
implementation.  I was considering adding empty functions to the base
classes as placeholders for derived code & possibly as a place for the
documentation.

Cheers,

Malcolm

[Lxr-developer] Doco template

[Sethna, Z. (R. Brisbane) <ZS...@rs...>]

HI

I want to propose that we us the following file header and function template
(in perl's POD format) to document the LXR source code. Tried to keep it
simple as possible.


# File Header
=pod

=head2 SYNOPSIS

        Purpose of functions in this file

=head2 FUNCTIONS

=cut
# End File Header


# Function Header
=pod

=head3 function name

   Desciption of function

=cut
#End Function Header


html documentation can then be generated by doing:

pod2html pod2html --title="LXR Source - genxref" --infile=filename
--outfile=filename.html --header

I'll write a perl script to call this command for each perl file.

Have a look at http://www.zeta.org.au/~nbsethna/genxref.html to see an
(empty) example of the html output.


II'll be away next week so I won't be able to reply to any email till the
following week, after which I shall be producing html pages earnestly ;-)


Cheers


Zubin

Re: [Lxr-developer] Weird MySQL bug

[<pe...@kl...>]

Malcolm Box <ma...@br...> writes:

> $dbh, which is the database handle in question, is a package
> variable and therefore I think it should be scoped until the package
> is removed, ie when the END block deletes things properly.  

I would tend to agree with you, FWIW.  I don't use mysql myself, so I
can't experiment, but have you tried

 * Setting $dbh->trace(n) with n>0 right after you connect?

 * Explicitly qualifying $dbh with the package, ie
   s/$dbh/$LXR::Index::Mysql::dbh/ ?

 * A different version of perl?

Not that I would know what was wrong given results of those tests, but
maybe they can help shed light on the problem?

...Peder...
-- 
Cogito ergo panta rei.

[Lxr-developer] Weird MySQL bug

[Malcolm B. <ma...@br...>]

Does anyone want to take a look at this:
https://sourceforge.net/tracker/index.php?func=detail&aid=428513&group_id=27350&atid=390117
bug?

It is about warning messages about destroying a database handle before
disconnect, and I can't for the life of me see what's causing it.  $dbh,
which is the database handle in question, is a package variable and
therefore I think it should be scoped until the package is removed, ie
when the END block deletes things properly.  But something is obviously
going wrong.

Any perl experts out there with some time?

Malcolm

Re: Re[2]: [Lxr-developer] Code documentation

[Neil S. <ns...@ci...>]

I know mngosearch has been suggested before now:

http://search.mnogo.ru/

...as has htdig:

http://www.htdig.org/

However, whilst trying to locate the mngo URL, I chanced across the
following page which also lists a number of other free search/indexing
engines:

http://twiki.org/cgi-bin/view/Codev/SearchAttachments

Cheers,

Neil.


----- Original Message -----
From: "Rusty Carruth" <rca...@te...>
To: <lxr...@li...>
Cc: <ns...@ci...>
Sent: Wednesday, June 13, 2001 3:43 PM
Subject: Re[2]: [Lxr-developer] Code documentation


> "Neil Salter" <ns...@ci...> wrote:
> > Hi Malcom,
> >
> > ...
> > I'd also - humbly - suggest that replacing glimpse with a free engine
should
> > be a higher priority ...
>
> I'm using an older glimpse version just because its free - what are the
alternatives???
>
> thanks!
>
> rc

Re[2]: [Lxr-developer] Code documentation

[Rusty C. <rca...@te...>]

"Neil Salter" <ns...@ci...> wrote:
> Hi Malcom,
> 
> ...
> I'd also - humbly - suggest that replacing glimpse with a free engine should
> be a higher priority ...

I'm using an older glimpse version just because its free - what are the alternatives???

thanks!

rc

Re: [Lxr-developer] Code documentation

[Neil S. <ns...@ci...>]

Hi Malcom,

Sorry about the delayed reply, I've been busy...

> I too have been considering this, since on the projects I'm working on
> we use both LXR and Doxygen, and their both served from the same tree.
>  Looking at that URL you gave, they've done the reverse thing, which is
> make Doxygen link to LXR, it looks like we need to do the LXR -> Doxygen
> linkage.
>
> Here's what I think the features I'd like to see are:
>
> 1) From any LXR ident query, go to the associated doxygen documentation
> 2) Link to the doxygen file docs for each file in source view (or
> include at the top of the file)
> 3) Have the ident view display the doxygen usage/collaboration diagrams.
>
> Somehow this all needs to be configurable - Doxygen puts out a static
> tree of files, so we need a way of mapping from an identifier to the
> right place in the tree, in the presence of multiple definitions of the
> same symbol.   What I mean is that there is unlikely to be a one to one
> mapping between an identifier and a piece of documentation - more like a
> fuzzy n-to-n match.
>
> Anyone got any bright ideas how to make this work?

Yes it looks a bit more complicated than I first thought. I'm afraid I'm not
familiar enough with the workings of LXR to make any sensible comment. I
would say that this is a feature that could be left for later, though.
Whilst useful, I'd say it's more important to get installation
documentation, and the core setup stuff working first.

I'd also - humbly - suggest that replacing glimpse with a free engine should
be a higher priority than a doxygen link-up: users can manually look up
doxygen classes etc. in just a few clicks.

I feel a bit cheeky asking for these things without putting in any effort to
deliver them, but I'm mailing in on the principle that any feedback is
useful.

Cheers,

Neil.

Re: [Lxr-developer] Code documentation

[Malcolm B. <ma...@br...>]

Hi Neil,

Neil Salter wrote:

>Hello,
>
>The guys here: http://unstable.elemental.com/mozilla/ had the idea of cross
>linking between doxygen and LXR. I think this is a good idea. Currently I'm
>using doxygen to document a (fairly large) source tree, and I plan to use
>LXR on the same tree. I'd say it'd be pretty useful to be able to jump from,
>say, a classname in LXR to the doxygen page for the class - and vice versa.
>It could all be done manually by the user just now, but it might be worth
>thinking about as potential future feature for incorporation into the
>software.
>
I too have been considering this, since on the projects I'm working on 
we use both LXR and Doxygen, and their both served from the same tree. 
 Looking at that URL you gave, they've done the reverse thing, which is 
make Doxygen link to LXR, it looks like we need to do the LXR -> Doxygen 
linkage.

Here's what I think the features I'd like to see are:

1) From any LXR ident query, go to the associated doxygen documentation
2) Link to the doxygen file docs for each file in source view (or 
include at the top of the file)
3) Have the ident view display the doxygen usage/collaboration diagrams.

Somehow this all needs to be configurable - Doxygen puts out a static 
tree of files, so we need a way of mapping from an identifier to the 
right place in the tree, in the presence of multiple definitions of the 
same symbol.   What I mean is that there is unlikely to be a one to one 
mapping between an identifier and a piece of documentation - more like a 
fuzzy n-to-n match.

Anyone got any bright ideas how to make this work?

Cheers,

Malcolm

Re: [Lxr-developer] Code documentation

[Malcolm B. <ma...@br...>]

Hi Zubin,

Sethna, Zubin (RSA, Brisbane) wrote:

>I don't think it directly supports perl but I think it can be tweaked to be
>useable. Will try this unless someone can suggest a better/more perl
>friendly documentation tool.
>
>Zubin
>
Well, there's always POD, perl's native documentation format.  There's 
plenty of tools around to turn that into HTML or man pages as needed, so 
perhaps POD would be a better bet.

I'm happy with either - the main thing is to have the 
comments/documentation :-)

Malcolm