RE: [GD-General] Compile times

[Brian H. <bri...@py...>]

> I'm totally impressed. When you say complete build times, that means
> *all* the code involved in Quake 2 or Quake 3? All the DLLs, 
> everything? Was there some trick you were using to get such 
> incredible build times (other than using straight C)?

If I remember correctly, yes.

Okay, Quake 2 is open sourced, so I just tested it out.  On my very
modest 2xP3/933 w/ 512MB RAM using MSVC 6, compiling ref_gl, game, and
client, it took 37 seconds.

Let me repeat: 37 seconds.  

Compiling each component averages 10-15 seconds for a full rebuild.  If
editing a single source file, compile-edit-run is probably 3 seconds.

The trick was careful engineering.  Nothing mind boggling, just doing
the obvious stuff.  <windows.h> was limited to only the files that
needed it.  We used ANSI C, not straight C++.  But even so, if you don't
use templates heavily or STL, straight C++ can compile very quickly as
well.  Dependencies between modules were well guarded -- changes to the
renderer data structures had no effect on the game source code.  Changes
to the network stuff had no effect on the renderer, etc.

There was absolutely nothing magical about it, and that's why I'm
honestly confused and addled by the seeming necessity today to spend 10+
minutes to rebuild games on top notch hardware.  And most new games
today use scripting languages, so the build times should be close to
non-existent (Q2 had no scripting language).

Quake 3 was not significantly more complex, unless the Q3VM stuff that
John did ended up taking a lot of time, but I highly doubt that as well.

> We're using C++ and STL, and even though we have a good 
> physical organization (like Lakos described), compiling all 
> our libraries and game code for one platform could take up to 
> 15-20 minutes. That's only for a full build; a quick change 
> somewhere still requires building that file and then doing a 
> link, which is about 30-40 seconds. 
> 
> I find 30-40 seconds for a trivial change close to 
> unbearable. Especially when it takes another 10 seconds for 
> the game to start and another 10 seconds for the level to load.

The above is exactly what I mean.  STL is the devil's tool, I stand very
firmly on that.  I completely dislike the way C++ has made templates,
which are effectively glorified search and replace macros.  It leads to
code bloat, and changing any of your template code causes massive
cascades of rebuilds.

> What are other people doing to reduce their compile times? 
> Getting rid of STL is not really an option. It's way too 
> useful to replace it with a crippled, custom 
> container/algorithm set that is probably not even type safe. 

I just don't buy this one second.  We survived and thrived just fine
before STL.  STL is very, very convenient, but is it worth that cost?  I
don't think so, not one bit.  

Writing basic data structures like sets, lists, hashes, etc. is stuff we
all did as undergrads in college.  In the massive, grand scheme of
engineering, it's a non-issue.  It's tedious, sure, but even then, that
tedium is measured in minutes or hours, not days and weeks.  The
overhead from using STL or any other features that incur extensive build
times affects you through the life of the project.  Every compile.

This, to me, is one of the biggest indicators that software engineering
has gotten completely out of hand.  And I'm not coming down on you
specifically, because pretty much 90% of all game companies that I'm
aware of are using STL or something similar in such a capacity, and
incurring the costs.  Yet I rarely see any potential benefits measured
concretely by this.

Probably the #1 defense for STL is that it prevents you from doing
something lame just to get things going.  The classic case is, say,
hardcoding an array when you should have a proper container class.  Or
using a linear search when a proper set/bag/map should be used.  But 20
minutes per rebuild (and in many cases, much much more from what I've
heard) to make up for lack of programmer discipline seems ridiculous.

I use linear searches.  And when I know they're going to be a problem, I
substitute something better.  I usually put in a comment like:

//fixme: don't do a linear search here!

But it lets me get something implemented, and then I can substitute it
later on demand.

This isn't a case of NIH, which is the common counterargument when I
rant about STL.  I'm 100% for code reuse, and in fact use a ton of open
source libraries in my own code -- Lua, libpng, ijg, ogg vorbis, SDL,
etc.  But when code reuse has a measureable cost on productivity without
an associated benefit, I have to raise an eyebrow.

I just remain 100% unconvinced that STL (or almost any library or
language feature) is worth the loss of hours each day in build times.

Anyway, enough lecturing on that =)

The only other things you can do is go through with heavily analyze
every single dependency you have and try to minimize them.  A common
enough thing is to revert to pure dynamic allocation so that changing
the physical structure of a class won't force rebuilds by clients of
that class, i.e. the classic:

//publicly export this
class FooInterface
{
   public:
      virtual void interfaceFunc( void ) = 0;
};

//only seen by the Foo module
class FooImplementation : public FooInterface
{
  ...
};

This can hurt though, since I personally abhor excessive dynamic memory
allocation.  I much prefer to have my data structures pre-allocated as
much as possible, even though this does cause a much greater dependency.

-Hook