[Lxr-dev] [ lxr-Bugs-1691391 ] genxref/swish-e/binary files is VERY inefficient

[SourceForge.net <no...@so...>]

Bugs item #1691391, was opened at 2007-03-30 16:55
Message generated for change (Settings changed) made by mbox
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1691391&group_id=27350

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: genxref
Group: current cvs
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Paul D. Smith (psmith)
>Assigned to: Malcolm Box (mbox)
Summary: genxref/swish-e/binary files is VERY inefficient

Initial Comment:
I have a number of very large binary files (cross-compilers, etc.) in my source trees, and when I genxref (using swish-e) it sucks up way more CPU/memory/time than I would expect.  I examined the code and found an algorithmic error.

For swish-e, genxref reads the contents of the files into memory (in Perl), THEN it examines the file to determine whether it's binary or not using File::MMagic::checktype_contents().  If the file is determined to be binary (or empty) we skip it.

This is obviously extremely inefficient, since File::MMagic never needs to see the entire file to know its file type: it needs to look at the filename (which it doesn't have here!) and possibly the first 8K or so of the file.

So, I rewrote this section of genxref to get a filehandle for the file, then test to see if it's binary, and only if it's not do we read the contents of the file into memory.  This sped up my genxref significantly and greatly reduced the load on my server during indexing.


Patch from latest CVS attached.

----------------------------------------------------------------------

>Comment By: Malcolm Box (mbox)
Date: 2009-03-23 16:51

Message:
Thanks for the patch, have applied this to CVS.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=390117&aid=1691391&group_id=27350