[Lxr-dev] [ lxr-Bugs-506627 ] Excessive memory usage

[<no...@so...>]

Bugs item #506627, was opened at 2002-01-21 12:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350

Category: genxref
Group: current cvs
Status: Closed
Resolution: Fixed
Priority: 4
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Malcolm Box (mbox)
Summary: Excessive memory usage

Initial Comment:
LXR uses much more memory than is required. I have a 
large binary with debugging symbols (80+M) in my 
source tree and attempting to index it I get the 
following error.

Out of memory during "large" request for 134221824 
bytes, total sbrk() is 203704320 bytes at 
lib/LXR/Lang.pm line 42.

Note the first value is equal to 128M. A quick look at 
the source code shows the offending line below:

# Try to see if it's a script
$files->getfile($pathname, $release) =~ /^#!\s*(\S+)/s;

And the offending function from Plain.pm:

sub getfile {
        my ($self, $filename, $release) = @_;
        my ($buffer);
        local ($/) = undef;

        open(FILE, $self->toreal($filename, $release)) 
|| return undef;
        $buffer = <FILE>;
        close(FILE);
        return $buffer;
}

Reading the whole file into memory to determine if it 
is a shell script is really excessive. Surely the 
first line should suffice. File::Type may do a better 
job, but it operates only on a filename, and I'm not 
sure about portability.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2002-02-05 10:30

Message:
Logged In: NO 

I'm using Plain files with Mysql. I'm also running it under 
cygwin. (FYI, I had to disable the copying source files 
to /tmp during the reference phase, since cygwin sometimes 
has problems creating files in /tmp, but this is not 
specific to LXR).

I'd be willing to add some tracing info, but I'm not 
familiar with the Devel::Leak package, so please tell me 
where and what to add.

----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2002-02-05 02:37

Message:
Logged In: YES 
user_id=215386

Hmm, that level of memory usage is strange.  I regularly
index a multi-gigabyte source tree with no memory problems. 
The tree doesn't contain any large binary file, but that
really shouldn't cause any problems.

What setup are you using - CVS/Plain files, Mysql/Postrgres
db?  If you could add some Devel::Leak tracing as well that
would be excellent...

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2002-01-23 10:15

Message:
Logged In: NO 

> Keeping binaries in a source tree is rarely a good thing.

Preaching to the choir...
But the fact of the matter is that large files often do end
up in source trees (trace files, log files, docs ...) just
because it helps for sharing and organization.

Even with this bug fixed, memory usage seems excessive. I
left LXR running for about a day, and found it had slowed to
a crawl because it was swapping constantly. Each perl
process was using about 200 M. It crashed the following day
when it ran out of memory.  I don't know what caused this
yet (I just started it up again).

----------------------------------------------------------------------

Comment By: Malcolm Box (mbox)
Date: 2002-01-22 23:58

Message:
Logged In: YES 
user_id=215386

I suppose one answer would be "Don't do that then" :-) 
Keeping binaries in a source tree is rarely a good thing.

In terms of fixing this, the easiest way would be to use the
Files::getfilehandle rather than Files::getfile to read the
first line.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=390117&aid=506627&group_id=27350