Thread: [Java-gnome-developer] Build time performance boosts

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

[long email, but worth it - please read]

++

Mark rightly points out that there are two modes:

  1) build the whole damn thing, from scratch

  2) compile just what I have updated, because I'm hacking on it.

I have some thoughts on the subject:

++

Having grown up in the grand old Unix tradition, I have always
understood Makefiles, and the rational for such a program: don't want to
recompile everything just because of one little change. So, you'd have

        blah: blah.o foo.o bar.o bling.o
        	ld -o blah blah.o foo.o bar.o bling.o

        blah.o: blah.c
        	cc -o blah.o -c blah.c

        foo.o: foo.c
        	cc -o foo.o -c foo.c
        ...

and so on. Of course, if you were compiling from scratch, it'd do
everything which was good, and if you made a single source change, you'd
only get a single compile and a single link.

For the longest time, I always thought that this was good.

++

In recent years, I've started to wonder about this. For example, 99% of
the time, when I compile something, it's not software I'm developing but
some package I'm building to install on my system. And so I sit there
and watch endless cc commands (each with two pages of -I flags) scroll
by.

The economics of CPU, memory, and the nature of the language have
changed. This is why languages like python (and Java, for that matter)
are feasible - the overhead of an interpreter (+ VM, etc) is not
terribly significant given today's available processing power.

It's not so obvious with gcc, but with javac, oh man.

Because it runs in a Java VM, javac takes a fair while to fire up. Java
source files themselves tend to be very small. There is a lot of
repetitive lookups to check references in all the imports. So you do
this long process for each of thousands of .java files, and you're there
a while.

++

Now of course, we've pointed out...

(I thought `ant` had something to do with it, but of course, that's just
ant doing a single javac invocation. I stand corrected)

...that javac actually has a lot of smarts in it - in fact, you could
hand javac nothing but your top level Main.java and it would probably
reach out and compile your entire application.

[for a long time I didn't trust that, but then I noticed it doing it for
me and covering my ass when I'd in fact made a Makefile mistake, so now
I feel better about it!]

So I started thinking about what would happen if reworked my own
projects' make files to just do one single javac invocation (to build
the .class files) and one single gcj invocation (straight to target .so
or executable) with all my .java files as input - regardless of whether
or not they are new.

Of course, I was horrified at the thought of getting it to compile
things it didn't have to, but

<speculation>
        the effort of compiling, and the effort of looking up the
        classes for import  is about the same regardless, and the gain
        would be not doing n invocations for n files
</speculation>

So I tried it.

Wow. Fast fast fast! Much faster than waiting for the individual gcj
and/or javac calls to scroll by. So fast, in fact, that a rebuild after
a trivial change to one file was still faster on the monolithic model
than on the one-by-one model.

++

So in view of this week's conversation, I decided to try it with
libgtk-java.

With my system otherwise idle, libgtk-java takes this long to build
(this is `make` in a clean (but configured) top level directory):

        real    17m40.494s
        user    14m49.584s
        sys     1m8.221s

[This box is a dual P3; my laptop takes over 40 minutes :(]

I then did some quick hacks to libgtk-java/src/Makefile to replace the
individual invocations of javac (.class files) and gcj (.o files) and
gcc (JNI .o files) with single steps

[ie a single javac with all the source files listed, a single gcj
command going straight to the .so with all the .java source files
listed, and a single gcc command going straight to the jni .so file with
all the .c files listed]

        real    4m37.743s
        user    4m11.783s
        sys     0m22.127s

Wow!

The amazing thing is that the slowest part of all is the jni C
compiling! Take that out, and the time to compile the java source (both
javac and gcj) was only

        real    1m5.500s
        user    1m1.663s
        sys     0m1.988s

that's 65 seconds!

Wow!

++

I've attached the simple diff (and again, in patch form) that shows the
makefile differences. As I said, all it does is loose the individual
compile steps.

I'm not sure that this is a good idea, overall...

obviously, if you're hacking on the bindings then you don't really want
to wait even 4 minutes for a recompile. But If you've already built it,
and you're only working on the Java side of things, then 1 minute isn't
so bad for a full rebuild [shit, you probably have to make install
before you test it anyway]

...but 4 minutes beats the hell out of 18 minutes if you're just an end
user building the package for installation on your system.

Food for thought.

AfC

-- 
Andrew Frederick Cowie

OPERATIONAL DYNAMICS
Operations Consultants and Infrastructure Engineers

Australia: +61 2 9977 6866  North America: +1 646 472 5054

http://www.operationaldynamics.com/

Thread: [Java-gnome-developer] Build time performance boosts

java-gnome-developer