lxr-developer — Don't use, active developers only

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Thu, 2007-03-22 14:29:24 -0400, Paul Smith <ps...@ne...> wrote:
> The only real difference I see is in handling the auto-increment fields:
> I don't know how Oracle does this but MySQL and Postgress handle this
> very differently.  So, that bit would need to be kept local.

Erm, I'm not 100% sure, but as far as I remember, the only difference
for "auto_increment" or "serial" or whatever fields is the CREATE
TABLE statement, at least for MySQL vs. PostgreSQL. Should be enough
to simply omit the auto_increment column's data to get a value
generated.

What may be different is the way to get the newly auto-generated
number back again. But since the INSERT pathes aren't time-critical,
we'd just allow to SELECT for the value instead of playing tricks to
get it.

> > The cruelity of MySQL is that it seems to require '`' instead of '"',
> > what all other RDBMS servers seem to accept...
>=20
> True :-(.  This could be the death of the commonality of the prepare()
> statements (although we could still share at least some of the methods
> since they just invoke execute()).

Heck, this f*ing column name caused so much grief, lets just rename
it!  Yes, that's somewhat painful and we need a Big Fat Warning in the
v1.0 docs that the column needs to be renamed, but I'm all for doing
that.

> Actually there is an "ANSI_QUOTES" mode that we could set, that lets you
> (among other things) use standard "-quoting in MySQL.  That might be a
> valid thing to do.

Is this in the DBI backend or configury on the MySQL server side?

> > The (current) initdb scripts try to first drop table foo, then CREATE
> > it.  Databases implemented different dialects to avoid errors (which
> > in turn could break a transaction), like the "drop table foo if
> > exists" stuff...
>=20
> True.  Note I wasn't suggesting that we would try to unify the initdb-*
> scripts into one script: there's too much variation in the schema
> creation operations to make that worthwhile IMO.

But wrapping around somehow should be done to ease creating tables
with prefixes.

> I just wanted a way to create databases and tables based on the
> information in lxr.conf, without having to edit the appropriate initdb-*
> script by hand.

ACK.

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:           Ich hatte in letzter Zeit ein bi=C3=9Fchen viel Rea=
litycheck.
the second  :               Langsam m=C3=B6chte ich mal wieder weitertr=C3=
=A4umen k=C3=B6nnen.

[Lxr-dev] LXR: Overview over recent patches and improvement ideas

[Jan-Benedict G. <jb...@lu...>]

Hi!

Since I was about to loose overview over the patches and suggestions,
I've pushed the recent stuff into a Wiki page.

Feel free to add your stuff, I probably forgot about a zillion more
important fixes.


	http://lug-owl.de/LugWiki/LXR


Where it says "Email not found", I couldn't find the email in SF.net's
email archive, but only in my mailbox. I've put a link to the
Message-ID in there for these cases...

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:         Alles wird gut! ...und heute wirds schon ein bi=C3=9F=
chen besser.
the second  :

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Thu, 2007-03-22 at 17:52 +0100, Maximilian Wilhelm wrote:
> > Critical to get in, IMO, are the enhancements to allow the code to
> work
> > with MySQL 5 (the quoting of `release` so it's not considered a
> > keyword).
>=20
> Would it be problem to quote it every time?

It's not a problem to quote it, EXCEPT THAT we then cannot combine the
backends because (as Jan-Benedict points out below) MySQL doesn't by
default accept standard SQL quoting chars.  This can be addressed, but
it might be simpler to just rename the column instead.

> I guess these scripts would need the 'baseurl' as parameter to get the
> correct data from lxr.conf?

Ooh, good point.

> The use case would be to have several project databases indexed by lxr
> side by side in the same database but with different table
> prefixes/names?

> I=B4m not that sure if this is really needed, maybe others have some
> experiences about that?

It's helpful for testing, at the very least.  Also I'm sure there are
sites that have multiple completely different codebases they'd like to
use with a single instance of LXR.

The package does give people the option of using an alternate database
names and/or table prefixes; it just seems odd that this configurability
is there in the code but not available at the table creation time.

Jan-Beneict Glaw <jb...@lu...> wrote:
> About the first paragraph:  I haven't had a look at the separate
> backends for PostgreSQL, MySQL or Oracle.  But I guess there actually
> was a reason to generate a whole class for implementing different
> backends instead of just using plain DBI.

I'm not so sure there was a reason, other than maybe an unfamiliarity
with Perl class inheritance and/or DBI itself.  Looking at the Index
backends I can't see any _material_ difference between Postgres and
MySQL (I haven't looked at Oracle).  They are pretty different in some
ways but that difference feels to me more like someone just updated one
backend but didn't bother with the other one (e.g., the MySQL backend
keeps the prepared statements inside the object, while Postgres uses
class variables).  If I had to guess based on the code I'd guess that
one of the backends was written, then when someone wanted a different
database the split was created in a more or less ad hoc way.

> The third paragraph stresses an assumption I have. There could be
> "prepared statements" inside the individual backends. When you submit
> something as simple as "select foo, bar from baz where xay=3D123", the
> DB server parses the request and tries to optimize it.

> A "prepared statement" allows us to declare that we'll use a specific
> query several times, but with different values. Even while the actual
> values differ, the way the tables are searched doesn't. This may save
> a lot of time.

Yes, unquestionably we should keep the prepared statement implementation
as we have it today.

> As I guess the backends implement prepared statements *as they are
> available* in the backends, we'd find a generic way to come up with
> prepared statements, or we'll have to keep the different backends.

The DBI interface takes care of all this for us, though: it supports the
prepare() method and does the best it can with whatever the underlying
database supports.  At the very least it saves the template in the
statement object and binds the values when you run execute().

In other words, almost all those prepare statements are identical across
all the backends, because they're all pure SQL using the same DBI
interface; they can all be pushed up into a superclass, along with most
of the methods.

The only real difference I see is in handling the auto-increment fields:
I don't know how Oracle does this but MySQL and Postgress handle this
very differently.  So, that bit would need to be kept local.

> The cruelity of MySQL is that it seems to require '`' instead of '"',
> what all other RDBMS servers seem to accept...

True :-(.  This could be the death of the commonality of the prepare()
statements (although we could still share at least some of the methods
since they just invoke execute()).

Actually there is an "ANSI_QUOTES" mode that we could set, that lets you
(among other things) use standard "-quoting in MySQL.  That might be a
valid thing to do.

> The (current) initdb scripts try to first drop table foo, then CREATE
> it.  Databases implemented different dialects to avoid errors (which
> in turn could break a transaction), like the "drop table foo if
> exists" stuff...

True.  Note I wasn't suggesting that we would try to unify the initdb-*
scripts into one script: there's too much variation in the schema
creation operations to make that worthwhile IMO.

I just wanted a way to create databases and tables based on the
information in lxr.conf, without having to edit the appropriate initdb-*
script by hand.

--=20
---------------------------------------------------------------------------=
--
 Paul D. Smith <ps...@ne...>                       http://netezza.co=
m
 "Please remain calm--I may be mad, but I am a professional."--Mad Scientis=
t
---------------------------------------------------------------------------=
--
      These are my opinions--Netezza takes no responsibility for them.

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Thu, 2007-03-22 12:37:04 -0400, Paul Smith <ps...@ne...> wrote:
> On Thu, 2007-03-22 at 17:06 +0100, Jan-Benedict Glaw wrote:
> > > Critical to get in, IMO, are the enhancements to allow the code to wo=
rk
> > > with MySQL 5 (the quoting of `release` so it's not considered a
> > > keyword).  A number of the bugs and patches on the Savannah site all
> > > deal with this single issue.
> >=20
> > That may need a fix.  If the DB backend gets rewritten to make deeper
> > use of the DBI interface (eg. don't do too much database specific
> > stuff), that'd hopefully be dropped altogether.
> >=20
> > Renaming the field to avoid the issue could help, too.
> >=20
> > And a Perl guru could have a look at the documentation to tell us if
> > DBI has a generic interface for prepared statements.
>=20
> The only thing I understood completely here was the second paragraph :).

About the first paragraph:  I haven't had a look at the separate
backends for PostgreSQL, MySQL or Oracle.  But I guess there actually
was a reason to generate a whole class for implementing different
backends instead of just using plain DBI.

The third paragraph stresses an assumption I have. There could be
"prepared statements" inside the individual backends. When you submit
something as simple as "select foo, bar from baz where xay=3D123", the
DB server parses the request and tries to optimize it.

A "prepared statement" allows us to declare that we'll use a specific
query several times, but with different values. Even while the actual
values differ, the way the tables are searched doesn't. This may save
a lot of time.   As I guess the backends implement prepared statements
*as they are available* in the backends, we'd find a generic way to
come up with prepared statements, or we'll have to keep the different
backends.

> As far as I'm aware there's no way to "make deeper use of the DBI
> interface" that would allow us to do something like ignore column
> names... ?  And I'm not sure what you mean by a generic interface for
> prepared statements.

A column name must be given. The cruelity of MySQL is that it seems to
require '`' instead of '"', what all other RDBMS servers seem to
accept...

> Hm... maybe you mean using placeholders ("?") for column names as well
> as for values, as in:
>=20
> 	select ? from lxr_releases where ? =3D ?
>=20
> If so, that won't work.  DBI supports placeholders ONLY for values, not
> for table/column/database names or other keywords.

You can, in theory, use 'select * from ....', but if you're forced to
manually fiddle with the table at some given time, you cannot be sure
that the order of colums will be stable.  The where clause will of
course also give you grey hair with `release` ...

> > The interesting part here is most probably to correctly deal with
> > dropping the old tables...
>=20
> Which old tables would we want to drop?


The (current) initdb scripts try to first drop table foo, then CREATE
it.  Databases implemented different dialects to avoid errors (which
in turn could break a transaction), like the "drop table foo if exists"
stuff...

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:                   ...und wenn Du denkst, es geht nicht mehr,
the second  :                          kommt irgendwo ein Lichtlein her.

Re: [Lxr-dev] Status of this project?

[Maximilian W. <ma...@rf...>]

Am Thursday, den 22 March hub Paul Smith folgendes in die Tasten:

> On Thu, 2007-03-22 at 16:24 +0100, Jan-Benedict Glaw wrote:

> > That said, I think we should have a view at the documentation again.
> > After that, lets release v1.0 and focus on the new goodies afterwards.

> That sounds fine, but maybe a 0.9.5 release for people to test before
> 1.0?

I think that´s a good idea.

> > > I´ve a yet unfinished patch to push the version and arch lists
> > > in drop-down boxes but this breaks the diff function IIRC.

> > I think this should be v1.1 material.

> Agree.

More time to hack on it :)

> > The speed-up patches posted could probably make it into v1.0, though.

> The second version of the patch, that doesn't require changes to
> LXR::Files, I think is a good candidate.  It's a simple algorithmic
> change that only impacts genxref and clearly makes sense, and helps a
> lot when dealing with swish-e and trees with binary files.

Sound good.

> > There's also the exit() compatibility patch for perl5. I think this
> > should also go in before v1.0 is pushed out.

> Critical to get in, IMO, are the enhancements to allow the code to work
> with MySQL 5 (the quoting of `release` so it's not considered a
> keyword).  A number of the bugs and patches on the Savannah site all
> deal with this single issue.

Would it be problem to quote it every time?

> There needs to be some cleanup in this as well; note that the lxr.conf
> file allows you to rename the database and table prefix, which is
> great... but there's no way to CREATE those databases/tables with
> different names other than editing the initdb-* file by hand, which is
> not so great.

> Maybe the initdb-* files need to become Perl scripts, which can read
> lxr.conf and make the appropriate substitutions.  Then the invocation
> model could change to something like:

> 	./initdb-mysql | mysql
> 	./initdb-postgres | psql

I guess these scripts would need the 'baseurl' as parameter to get the
correct data from lxr.conf?

The use case would be to have several project databases indexed by lxr
side by side in the same database but with different table
prefixes/names?

I´m not that sure if this is really needed, maybe others have some
experiences about that?

> (not sure for Oracle).  I can look into that if people think it's a good
> idea.

Sound like an interesting approach.

Ciao
Max
-- 
	Follow the white penguin.

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Thu, 2007-03-22 at 17:06 +0100, Jan-Benedict Glaw wrote:

> > Critical to get in, IMO, are the enhancements to allow the code to work
> > with MySQL 5 (the quoting of `release` so it's not considered a
> > keyword).  A number of the bugs and patches on the Savannah site all
> > deal with this single issue.
> 
> That may need a fix.  If the DB backend gets rewritten to make deeper
> use of the DBI interface (eg. don't do too much database specific
> stuff), that'd hopefully be dropped altogether.
> 
> Renaming the field to avoid the issue could help, too.
> 
> And a Perl guru could have a look at the documentation to tell us if
> DBI has a generic interface for prepared statements.

The only thing I understood completely here was the second paragraph :).
Renaming the column is fine with me but would mean we need to advise
people who already have databases on how to use ALTER TABLE to fix their
schema.  Putting on my SCM hat, I'd suggest a good alternative name for
"release" is "stream".  Probably it's a good practice to have an
"upgrade" script that people would invoke to move from an older to a
newer version.

As far as I'm aware there's no way to "make deeper use of the DBI
interface" that would allow us to do something like ignore column
names... ?  And I'm not sure what you mean by a generic interface for
prepared statements.

Hm... maybe you mean using placeholders ("?") for column names as well
as for values, as in:

	select ? from lxr_releases where ? = ?

If so, that won't work.  DBI supports placeholders ONLY for values, not
for table/column/database names or other keywords.

> The interesting part here is most probably to correctly deal with
> dropping the old tables...

Which old tables would we want to drop?

-- 
-----------------------------------------------------------------------------
 Paul D. Smith <ps...@ne...>                       http://netezza.com
 "Please remain calm--I may be mad, but I am a professional."--Mad Scientist
-----------------------------------------------------------------------------
      These are my opinions--Netezza takes no responsibility for them.

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Thu, 2007-03-22 11:49:11 -0400, Paul Smith <ps...@ne...> wrote:
> On Thu, 2007-03-22 at 16:24 +0100, Jan-Benedict Glaw wrote:
> > That said, I think we should have a view at the documentation again.
> > After that, lets release v1.0 and focus on the new goodies afterwards.
>=20
> That sounds fine, but maybe a 0.9.5 release for people to test before
> 1.0?

Sure.

> > There's also the exit() compatibility patch for perl5. I think this
> > should also go in before v1.0 is pushed out.
>=20
> Critical to get in, IMO, are the enhancements to allow the code to work
> with MySQL 5 (the quoting of `release` so it's not considered a
> keyword).  A number of the bugs and patches on the Savannah site all
> deal with this single issue.

That may need a fix.  If the DB backend gets rewritten to make deeper
use of the DBI interface (eg. don't do too much database specific
stuff), that'd hopefully be dropped altogether.

Renaming the field to avoid the issue could help, too.

And a Perl guru could have a look at the documentation to tell us if
DBI has a generic interface for prepared statements.

> There needs to be some cleanup in this as well; note that the lxr.conf
> file allows you to rename the database and table prefix, which is
> great... but there's no way to CREATE those databases/tables with
> different names other than editing the initdb-* file by hand, which is
> not so great.

Right.

> Maybe the initdb-* files need to become Perl scripts, which can read
> lxr.conf and make the appropriate substitutions.  Then the invocation
> model could change to something like:
>=20
> 	./initdb-mysql | mysql
> 	./initdb-postgres | psql
>=20
> (not sure for Oracle).  I can look into that if people think it's a good
> idea.

The interesting part here is most probably to correctly deal with
dropping the old tables...

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:                http://catb.org/~esr/faqs/smart-questions.html
the second  :

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Thu, 2007-03-22 at 16:24 +0100, Jan-Benedict Glaw wrote:

> That said, I think we should have a view at the documentation again.
> After that, lets release v1.0 and focus on the new goodies afterwards.

That sounds fine, but maybe a 0.9.5 release for people to test before
1.0?

> > I=B4ve a yet unfinished patch to push the version and arch lists
> > in drop-down boxes but this breaks the diff function IIRC.
>=20
> I think this should be v1.1 material.

Agree.

> The speed-up patches posted could probably make it into v1.0, though.

The second version of the patch, that doesn't require changes to
LXR::Files, I think is a good candidate.  It's a simple algorithmic
change that only impacts genxref and clearly makes sense, and helps a
lot when dealing with swish-e and trees with binary files.

> There's also the exit() compatibility patch for perl5. I think this
> should also go in before v1.0 is pushed out.

Critical to get in, IMO, are the enhancements to allow the code to work
with MySQL 5 (the quoting of `release` so it's not considered a
keyword).  A number of the bugs and patches on the Savannah site all
deal with this single issue.


There needs to be some cleanup in this as well; note that the lxr.conf
file allows you to rename the database and table prefix, which is
great... but there's no way to CREATE those databases/tables with
different names other than editing the initdb-* file by hand, which is
not so great.

Maybe the initdb-* files need to become Perl scripts, which can read
lxr.conf and make the appropriate substitutions.  Then the invocation
model could change to something like:

	./initdb-mysql | mysql
	./initdb-postgres | psql

(not sure for Oracle).  I can look into that if people think it's a good
idea.

--=20
---------------------------------------------------------------------------=
--
 Paul D. Smith <ps...@ne...>                       http://netezza.co=
m
 "Please remain calm--I may be mad, but I am a professional."--Mad Scientis=
t
---------------------------------------------------------------------------=
--
      These are my opinions--Netezza takes no responsibility for them.

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Thu, 2007-03-22 14:40:33 +0100, Maximilian Wilhelm <ma...@rf...> wro=
te:
> Am Thursday, den 22 March hub Malcolm Box folgendes in die Tasten:
> > It's true we're woefully overdue for a release - CVS head is
> > reasonably stable and could arguably be made v1.0.  I've got a bunch
> > of things I want to experiment with in the browsing experience, but
> > they're post-1.0 features.  And require time...

That said, I think we should have a view at the documentation again.
After that, lets release v1.0 and focus on the new goodies afterwards.

> I=C2=B4ve a yet unfinished patch to push the version and arch lists
> in drop-down boxes but this breaks the diff function IIRC.

I think this should be v1.1 material.

The speed-up patches posted could probably make it into v1.0, though.
Malcom, what do you think?

There's also the exit() compatibility patch for perl5. I think this
should also go in before v1.0 is pushed out.

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:         Alles wird gut! ...und heute wirds schon ein bi=C3=9F=
chen besser.
the second  :

Re: [Lxr-dev] Status of this project?

[Maximilian W. <ma...@rf...>]

Am Thursday, den 22 March hub Malcolm Box folgendes in die Tasten:

Hi!

> It's true we're woefully overdue for a release - CVS head is
> reasonably stable and could arguably be made v1.0.  I've got a bunch
> of things I want to experiment with in the browsing experience, but
> they're post-1.0 features.  And require time...

I´ve a yet unfinished patch to push the version and arch lists
in drop-down boxes but this breaks the diff function IIRC.

After my exams I hopefully find some time to look at this again.

Ciao
Max
-- 
	Follow the white penguin.

Re: [Lxr-dev] Status of this project?

[Arne G. G. <ar...@gl...>]

Malcolm Box wrote:
> It's true we're woefully overdue for a release - CVS head is
> reasonably stable and could arguably be made v1.0.  I've got a bunch
> of things I want to experiment with in the browsing experience, but
> they're post-1.0 features.  And require time...

I, too, have a handful of experimental changes to LXR that are actually
starting to mature into something that might be useful for others.
Unfortunately (or not, as the case may be), since I find my own code
from 10 years ago both a bit baroque and embarrassing, much of the
framework in my tree has been refactored and reworked to such a state
that it's not readily compatible with mainline anymore.  If there's
interest, I'll but the code up somewhere for others to poke at.  There
might be stuff there that could be rolled back into mainline, and it
might even be interesting as a development base for more experimental stuff.

-- 
							Arne.

Re: [Lxr-dev] Status of this project?

[Malcolm B. <mal...@gm...>]

Hi,

On 3/18/07, Jan-Benedict Glaw <jb...@lu...> wrote:
> On Fri, 2007-03-16 11:38:34 -0400, Paul Smith <ps...@ne...> wrote:
> > Is anyone actively working on this project?  I see some commits went in
> > in January, but there are a lot of bug reports that aren't being managed
> > (many duplicate bugs and no one seems to be closing them) and a number
> > of the bugs being reported are already fixed in CVS, but there's no
> > release available, or comments about when a release might become
> > available.  There are also a few patches that look like they (or the
> > ideas they implement) could be helpful.
>
> AFAICT, there's no "active" work being done at this time. The last
> patch I saw popping up here on the list was some perl5 compatibility
> stuff for callin exit(), IIRC (which is still unreviewed.)

Don't be so hard on yourself - you merged a GIT backend a couple of months ago!

It's true we're woefully overdue for a release - CVS head is
reasonably stable and could arguably be made v1.0.  I've got a bunch
of things I want to experiment with in the browsing experience, but
they're post-1.0 features.  And require time...

>
> > I have done some work in my local version which has a big performance
> > increase for my source tree, using swish-e (it won't help everyone's
> > environment: it really helps with handling non-text files).
>
> Great. Care to send some patches for review?

Indeed!

> > Before I go too much farther with enhancements/fixes I'd like to have an
> > idea on the status of LXR; is it still viable?  Are there thoughts about
> > a new release (even just a 0.9.5, not 1.0)?  Is there a need/desire for
> > new developers on the team?  Or, is everyone just using their own
> > customized version and they're happy with that?

There's both a need and a desire.  While lots of people have their own
customised versions, there's still a steady stream of downloads and I
believe that there's plenty left to do that would be valuable to the
majority.

> LXR is, to my knowledge, also the only project in this area, that
> works. So that's a definitive answer (at least from my point of view)
> that you're right in starting hacking right here if something doesn't
> work for you, or that you place bug reports.

Totally agreed!
>
> In the case of bug reports, I'd suggest sending them to the list and
> not to SourceForge's bug tracker.  I don't think anybody pays too much
> attention to it....

I read the tracker - and am more likely to remember the bug if I ever
get time to fix things than in the mailing list.  You'll get more
response in the mailing list though - so both is a good option :-)

> > Should I start looking more seriously at alternatives such as OpenGrok,
> > etc.?

Of course not - heresy :-)

Cheers,

Malcolm

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Wed, 2007-03-21 at 07:21 +0100, Gregor Hartmann wrote:

> >Personally, I'd like to see the entire LXR::Files class reimplemented.
> >IMO, the right way to do that is to have a class that represents a
> >single file

> Sounds like a great idea to me. Some time ago I noticed that files are 
> copied even when they are in the filesystem already (not in cvs). This 
> could be cleanly ommited then also by accessing the file directly.
> Spechial care has to be taken then that the file is not modified and 
> most important not deleted at the end.

After looking further I also think there's also a lot of opportunity to
fix the Index class and subclasses: right now LXR::Index is essentially
empty, but the reality is that, because of Perl's DBI interface which is
common across databases, with only a few minor exceptions all the
database access for all different databases will be identical.  I think
all the content of the LXR::Index::* subclasses should be pushed up into
the LXR::Index class, and the subclasses left there only for
implementing any DB-specific extensions that are required.  This would
make it much simpler to support new databases and would ensure that any
enhancements made to one DB were available to all without any extra
work.

-----
Actually the patch I sent before relied on another change I made, that
wasn't included.  I'm including a new one that doesn't require any
changes to the LXR::Files class to support it.

I have a number of other cleanups and enhancements I can send.  I'm
including here a fix to the "not all .h files are indexed" issue I
mentioned the other day; after a night's sleep I was able to pretty
easily figure out where that cross-referencing was done.  My fix looks
up filenames in the database if they are not found via the current path
lookup methods.  If only one file is found that matches, a link to it is
added.  If more than one file is found that matches, a link to a search
for that filename is used as the href.  I only include the MySQL version
here but the Postgres one is the same.  I'm not sure about Oracle... we
might have to let someone who has a copy fix it up.

Both of these patches will have some fuzz because of other enhancements
I've made, but they should apply.

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Sun, 2007-03-18 at 17:04 +0100, Jan-Benedict Glaw wrote:
> > I've also found a few other issues, like not all my header files are
> > getting hyperlinked properly.
> 
> Only found the issue or also a fix?

Only the issue.  I spent about 30 minutes staring at the code and
scratching my head trying to figure out where the #include files are
linked for C/C++ and couldn't find it.  I found the generic.conf file,
etc. but I guess I just don't quite understand how it's all used and
there's not much in the way of documentation :-/.

What I see is that #include'd files are linked if they are in the same
directory as the source, but not otherwise.  Apparently the system
doesn't know how to find non-local headers.

To me it seems like a good way to resolve this would be to use the file
list generated by swish to locate headers.  If we wanted to be fancy we
could say that if more than one header of a given name was found, the
#include could link to a search for that header name so the user could
decide (since we don't have the compiler line to figure it out for
real).

I don't know much about Glimpse, because it's not free software.  If it
doesn't have anything like swish's filenames file we could have genxref
create one for us.

Re: [Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

On Sun, 2007-03-18 at 17:04 +0100, Jan-Benedict Glaw wrote:
> Great. Care to send some patches for review?

Sure; here's a patch.  It avoids reading the entire contents of the file
until after we know we're going to need it.  In my source tree I have a
number of very large binary files (cross-compilers and similar things),
and the old method of reading the entire contents of the file into
memory first was extremely wasteful of system resources, not to mention
very slow.

Personally, I'd like to see the entire LXR::Files class reimplemented.
IMO, the right way to do that is to have a class that represents a
single file, where you'd run something like:

    $file = LXR::File->new($path, $release);

and get back an object reference for that file.  Then you'd run things
like:

    $fh = $file->gethandle();
    $size = $file->getsize();
    $path = $file->getpath();

etc.  The really nice thing about this is that when $file goes out of
scope, there's a destructor invoked automatically by Perl that you can
use to clean up any open filehandles, temporary copies of files you had
to create from CVS or GIT, etc. etc.  Currently, that all has to be done
by hand which is a pain, not to mention error-prone.

I've done a LOT of Perl hacking over the years.  If people are amenable
to these kinds of changes I'd be happy to make them.

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Sun, 2007-03-18 17:44:40 +0100, Maximilian Wilhelm <ma...@rf...> wro=
te:
> Am Sunday, den 18 March hub Jan-Benedict Glaw folgendes in die Tasten:
> [...]
> > OpenGrok uses Java (thus has a requirement to a language that probably
> > needs non-free interpreters) while LXR just uses simple plain
> > Apache+PHP.
>=20
> s/PHP/Perl/

Argh. You're right of course :D

> [...]
> > Finally, OpenGrok needs a Tomcat (or equivalent) running, while LXR
> > just needs a simple Apache instance.
>=20
> What about a DB-Backend?

Ah, right, we need one of those. MySQL, PostgreSQL, Oracle. Forgot
something?  But I guess OpenGrok either needs one, too, or they're
having a number of ctags files, one for each release.

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:               Tr=C3=A4ume nicht von Dein Leben: Lebe Deinen T=
raum!
the second  :

Re: [Lxr-dev] Status of this project?

[Maximilian W. <ma...@rf...>]

Am Sunday, den 18 March hub Jan-Benedict Glaw folgendes in die Tasten:

Hi!

[...]
> OpenGrok uses Java (thus has a requirement to a language that probably
> needs non-free interpreters) while LXR just uses simple plain
> Apache+PHP.

s/PHP/Perl/

[...]
> Finally, OpenGrok needs a Tomcat (or equivalent) running, while LXR
> just needs a simple Apache instance.

What about a DB-Backend?

Ciao
Max
-- 
	Follow the white penguin.

Re: [Lxr-dev] Status of this project?

[Jan-Benedict G. <jb...@lu...>]

On Fri, 2007-03-16 11:38:34 -0400, Paul Smith <ps...@ne...> wrote:
> Is anyone actively working on this project?  I see some commits went in
> in January, but there are a lot of bug reports that aren't being managed
> (many duplicate bugs and no one seems to be closing them) and a number
> of the bugs being reported are already fixed in CVS, but there's no
> release available, or comments about when a release might become
> available.  There are also a few patches that look like they (or the
> ideas they implement) could be helpful.

AFAICT, there's no "active" work being done at this time. The last
patch I saw popping up here on the list was some perl5 compatibility
stuff for callin exit(), IIRC (which is still unreviewed.)

> I have done some work in my local version which has a big performance
> increase for my source tree, using swish-e (it won't help everyone's
> environment: it really helps with handling non-text files).

Great. Care to send some patches for review?

> I've also found a few other issues, like not all my header files are
> getting hyperlinked properly.

Only found the issue or also a fix?

> Before I go too much farther with enhancements/fixes I'd like to have an
> idea on the status of LXR; is it still viable?  Are there thoughts about
> a new release (even just a 0.9.5, not 1.0)?  Is there a need/desire for
> new developers on the team?  Or, is everyone just using their own
> customized version and they're happy with that?

For years (at least for five years or so), there's only one "release"
I'd tell people to download: CVS HEAD.  So you shouldn't take a
released tarball to start.  I also think that we'd place a new
release. At least there was the GIT backend added, which (taken that
the initial scope of LXR is to visualize Linux kernel source code) I
think is a somewhat important addition.

LXR is, to my knowledge, also the only project in this area, that
works. So that's a definitive answer (at least from my point of view)
that you're right in starting hacking right here if something doesn't
work for you, or that you place bug reports.

In the case of bug reports, I'd suggest sending them to the list and
not to SourceForge's bug tracker.  I don't think anybody pays too much
attention to it....

> Should I start looking more seriously at alternatives such as OpenGrok,
> etc.?

You may choose whatever tool fit your needs, but I'm not really
thrilled by OpenGrok. Just look at their web page
(http://www.opensolaris.org/os/project/opengrok/). The comparison to
other projects is totally bogus. As we'd say in German, they're
comparing apples to pears.

cscope/ctags look pretty poor compared to LXR and OpenGrok, but
they're focussing a totally different problem. It's just not their
business presenting web-pages with source code! Finally only look at
LXR vs. OpenGrok:

Definition search:	"the feature may be partly present" in LXR? What
			the heck?! I think they've been on crack when
			they wrote that...

History search:		LXR is /not/ a history browser. It's ment to
			give access to well pre-defined releases of
			source code. If you want history display, use
			ViewCVS or one of the other history displaying
			tools. Not our business!

Shows matching lines:	What's this feature, exactly? LXR can show
			diffs between releases, but honestly, I don't
			know what they mean here.

Hierarchical Search,
query syntax like AND,
OR, field:		LXR doesn't do searching itself. Not LXR's
			business. LXR is more flexible by using
			external search engines. Buy a Google
			appliance (yes, really, you can have your own
			Google!) and we'll manage to stuff the sources
			in there!  But well-working searching is not
			something that should be too deeply built into
			any tool, because there are better tools
			available...

Incremental update:	We do that.

Interface for SCM:	Erm, without interfaces to SCMs, we'd face a
			hard time displaying anything, right? So we
			don't have an SCM interface? Heck, I've been
			working on the wrong project :)

open source:		Huuuhuhh?! Anybody at home? LXR is GPL? I
			always thought GPL _is_ an Open Source
			license. No?

Individual file
download:		They're right. We don't have that. Probably
			because nobody ever thought it would be
			useful? Need it? No problem. That'd be easy to
			add.

Changes at directory
level:			I don't understand this. What's it exactly?

Multi language support:	Just a matter of placing customized template
			files. But right now, we only have the English
			version.



So that comparison really looks bogus to me. But they're not
mentioning some parts that are somewhat imprortant to me:

OpenGrok uses Java (thus has a requirement to a language that probably
needs non-free interpreters) while LXR just uses simple plain
Apache+PHP.

OpenGrok is licensed under CDDL, while LXR is GPL. So OpenGrok ships
with a GPL-incompatible license...

Finally, OpenGrok needs a Tomcat (or equivalent) running, while LXR
just needs a simple Apache instance.

So have a look at both projects and decide which of those serves your
needs best.

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
 Signature of:                      http://perl.plover.com/Questions.html
 the second  :

[Lxr-dev] Status of this project?

[Paul S. <ps...@ne...>]

Hi all;

Is anyone actively working on this project?  I see some commits went in
in January, but there are a lot of bug reports that aren't being managed
(many duplicate bugs and no one seems to be closing them) and a number
of the bugs being reported are already fixed in CVS, but there's no
release available, or comments about when a release might become
available.  There are also a few patches that look like they (or the
ideas they implement) could be helpful.

I have done some work in my local version which has a big performance
increase for my source tree, using swish-e (it won't help everyone's
environment: it really helps with handling non-text files).

I've also found a few other issues, like not all my header files are
getting hyperlinked properly.

Before I go too much farther with enhancements/fixes I'd like to have an
idea on the status of LXR; is it still viable?  Are there thoughts about
a new release (even just a 0.9.5, not 1.0)?  Is there a need/desire for
new developers on the team?  Or, is everyone just using their own
customized version and they're happy with that?

Should I start looking more seriously at alternatives such as OpenGrok,
etc.?


Cheers!

-- 
-----------------------------------------------------------------------------
 Paul D. Smith <ps...@ne...>                       http://netezza.com
 "Please remain calm--I may be mad, but I am a professional."--Mad Scientist
-----------------------------------------------------------------------------
      These are my opinions--Netezza takes no responsibility for them.

[Lxr-dev] [ lxr-Bugs-1670720 ] ident script fails with MySQL 5 backend

[SourceForge.net <no...@so...>]

Bugs item #1670720, was opened at 2007-02-28 02:45
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1670720&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Database interface
Group: current cvs
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Tyrel Datwyler (tyreld)
Assigned to: Nobody/Anonymous (nobody)
Summary: ident script fails with MySQL 5 backend

Initial Comment:
The ident script erroneously returns 0 declarations and 0 references when using a MySQL v5 backend. No errors are reported in the web server error log. The ident script works fine when using a MySQL v4 backend.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1670720&group_id=27350

[Lxr-dev] [ lxr-Bugs-1645267 ] problem with MySQL syntax in delete statement

[SourceForge.net <no...@so...>]

Bugs item #1645267, was opened at 2007-01-26 14:01
Message generated for change (Settings changed) made by darope
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1645267&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Database interface
Group: v0.9.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Florian Zschocke (darope)
Assigned to: Nobody/Anonymous (nobody)
>Summary: problem with MySQL syntax in delete statement

Initial Comment:
The genxref, when called with parameter --reindexall, generates MySQL errors when used with a MySQL 5.0 backend.

DBD::mysql::st execute failed: Unknown table 'lxr_indexes' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 334.
DBD::mysql::st execute failed: Unknown table 'lxr_useage' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 335.
DBD::mysql::st execute failed: Unknown table 'lxr_status' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 336.
DBD::mysql::st execute failed: Unknown table 'lxr_files' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 338.


The problem appeas to lie in the MySQL statements used for "delete_indexes", "delete_useage", "delete_status" and "delete_files" in Mysql.pm.

Between lines 114 and 131 the statements are defined as
"delete from ${prefix}indexes using ${prefix}indexes i, ${prefix}releases r where ...".
This is not working and according to the MySQL manual not correct syntax. Since aliases are used, the alias has to be used with DELETE, too. The correct sytnax (which works for me) would be:
"delete from i using ${prefix}indexes i, ${prefix}releases r where ..."

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1645267&group_id=27350

[Lxr-dev] [ lxr-Bugs-1645267 ] problem with MySQL syntax in delete statenebt

[SourceForge.net <no...@so...>]

Bugs item #1645267, was opened at 2007-01-26 14:01
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1645267&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Database interface
Group: v0.9.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Florian Zschocke (darope)
Assigned to: Nobody/Anonymous (nobody)
Summary: problem with MySQL syntax in delete statenebt

Initial Comment:
The genxref, when called with parameter --reindexall, generates MySQL errors when used with a MySQL 5.0 backend.

DBD::mysql::st execute failed: Unknown table 'lxr_indexes' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 334.
DBD::mysql::st execute failed: Unknown table 'lxr_useage' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 335.
DBD::mysql::st execute failed: Unknown table 'lxr_status' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 336.
DBD::mysql::st execute failed: Unknown table 'lxr_files' in MULTI DELETE at /usr/share/perl5/LXR/Index/Mysql.pm line 338.


The problem appeas to lie in the MySQL statements used for "delete_indexes", "delete_useage", "delete_status" and "delete_files" in Mysql.pm.

Between lines 114 and 131 the statements are defined as
"delete from ${prefix}indexes using ${prefix}indexes i, ${prefix}releases r where ...".
This is not working and according to the MySQL manual not correct syntax. Since aliases are used, the alias has to be used with DELETE, too. The correct sytnax (which works for me) would be:
"delete from i using ${prefix}indexes i, ${prefix}releases r where ..."

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1645267&group_id=27350

[Lxr-dev] [ lxr-Feature Requests-942911 ] Subversion Support

[SourceForge.net <no...@so...>]

Feature Requests item #942911, was opened at 2004-04-27 05:23
Message generated for change (Comment added) made by mengland
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390120&aid=942911&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: General
Group: Next Release
Status: Open
Priority: 5
Private: No
Submitted By: teewurstmann (teewurstmann)
Assigned to: Nobody/Anonymous (nobody)
Summary: Subversion Support

Initial Comment:
It would be nice to integrate support for Subversion
into a future release of LXR. Since LXR already
supports CVS support for Subversion shouldn't be too
hard to implement.

----------------------------------------------------------------------

Comment By: Matt England (mengland)
Date: 2006-12-24 07:48

Message:
Logged In: YES 
user_id=10503
Originator: NO

Is any Subversion-support activity underway?

If not, can anyone provide some pointers on how to make an appropriate
lib/LXR/Files/SVN.pm and anything else that's needed to make LXR parse a
Subversion repository?

-Matt

----------------------------------------------------------------------

Comment By: Vaish (vaish)
Date: 2006-04-24 10:19

Message:
Logged In: YES 
user_id=1508835

Is there any workaround for LXR to access subversion repository?

----------------------------------------------------------------------

Comment By: Roland Schwingel (rolandschwingel)
Date: 2004-10-20 03:58

Message:
Logged In: YES 
user_id=710704

Oh yes!!

This would be really cool to have... I am at present
searching for an application doing this...

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390120&aid=942911&group_id=27350

[Lxr-dev] incprefix and maps

[Jan-Benedict G. <jb...@lu...>]

Hi!

While searching for a bug in the GIT backend, I stumbled over two
things that weren't optimal for my setup, patched by this:

diff --git a/templates/lxr.conf b/templates/lxr.conf
index 08a35ab..bed75b7 100644
--- a/templates/lxr.conf
+++ b/templates/lxr.conf
@@ -113,7 +113,7 @@
=20
 		# These do funky things to paths in the system - you probably don't need=
 them.
 		'maps' =3D> {
-			'/include/asm[^\/]*/' =3D> '/include/asm-$a/',
+			'/include/asm/'	=3D> '/include/asm-$a/',
 			'/arch/[^\/]+/'       =3D> '/arch/$a/',
 		},
=20
@@ -202,7 +202,7 @@
=20
 		# where to look for include files inside the sourcetree.  This is used t=
o hyperlink
 		# to included files.
-		'incprefix' =3D> [ '/include', '/include/linux' ],
+		'incprefix' =3D> [ '/include' ],
=20
 		# Which extensions to treat as images when browsing.  If a file is an im=
age,
 		# it is displayed.


Currently, using 2.6.x Linux kernel versions, includes for generic
files (eg. asm-*/ioctl.h includes asm-generic/ioctl.h) are mishandled.
The generated link for the asm-generic/ioctl.h file will point back to
whatever is in $a right now.

The second chunk cuts "/include/linux" out of the include list.
Today's kernels don't use this path; all #include lines properly carry
the linux/ path component with them.

As I currently don't have a setup containing old kernels, could
anybody please test that?

MfG, JBG

--=20
      Jan-Benedict Glaw      jb...@lu...              +49-172-7608481
Signature of:                     Eine Freie Meinung in einem Freien Kopf
the second  :                   f=C3=BCr einen Freien Staat voll Freier B=
=C3=BCrger.

[Lxr-dev] [ lxr-Bugs-1616875 ] Directory with the name of its parent

[SourceForge.net <no...@so...>]

Bugs item #1616875, was opened at 2006-12-15 23:42
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1616875&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Browsing
Group: v0.9.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Directory with the name of its parent

Initial Comment:
In LXR 0.9.4, if a dirctory has the same name as its parent directory, LXR failed to display the content of the directory.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1616875&group_id=27350