XML was RE: [GD-General] RE: A portable preferences library

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

> I like to clarify this a bit more. In case of CS making such a faster
loader
> really is a HUGE amount of work. We have LOTS of different data to save.
By
> using XML we can generalize the loader/saver a lot and this makes our
> project a lot
> more maintainable. Making specific optimized binary loaders and savers
> for all
> our different plugins is a huge undertaking and it also means a lot more
> code
> to maintain.
>
> Greetings,
>
Before anyone rides right in and starts taking shots at what Jorrit is
saying, I'd like to say that when data with crystal space is compressed into
ZIP files, the load times are actually pretty good.

I used and experimented a lot with CrystalSpace's source code some
months back (and I'm sure CS has improved considerably since then) and it
was pretty good - even load times wise - by any standards (I say that as
someone whose architected a commercial 3D engine technology; used Gamebryo;
used the
Quake engine; and experimented recompiling and changing those chunks of
Serious Sam's source code that're available).

Also, at a different time - years back; I experimented extensively with a
huffman coder coupled with a dictionary based compression scheme (basically
chunks of LHArc's source code that were freely available) and even on a 486,
the disk rather than the CPU was the bottleneck - and the computation time
to uncompress a 20 MB disk file (with 32 MB of RAM) was around 15 % of the
time it'd take to load the same data.
And with XML like data (label value pairs), it was often possible to get
compression ratios comparable to the same file in a binary format.
With XHTML documents one could often get extremely decent compression
ratios.

I would therefore imagine that if someone really were interested in using CS
as their underlying engine, it may be better to spend energy writing an
optimized ZIP file loader/decompressor (if the existing one's are not good
enough - I personally would imagine that either you could find existing
decompressors that're good enough/fast enough OR you can find something
that'd fit your purpose- maybe even a commercially available linkable
library).

I would therefore request all of us here who're in this discussion to add a
"PS: I use crystal space" or a "PS: I dont use crystal space" or "PS: I have
never even taken a shot at compiling the CS code" - and chances are that
most of us wouldnt have even touched CS with a bargepole.

Also, I feel that the lessons the people who're working on it learnt with
crystal space are extremely relevant - and it is counter productive to just
debate these points in a way that tries to look for revolutionary
discoveries or extreme stupidities of the "oooh... I found a mount everest
at the bottom of the ocean" or a "oh my God! that is horrible" rather than
looking at CS as being an evolutionary shared learning experience. That
defeats the whole point of both this group and of CS.

I hope that helps...

Best Regards,
Nalin
----- Original Message -----
From: "Jorrit Tyberghein" <Jor...@uz...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 8:20 PM
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library

> Alen Ladavac wrote:
>
> >>The PlaneShift game (made with CS) has about 25 megabytes (uncompressed)
> >>of XML files (aggregated
> >>in four big files) of level data. On my Linux system at home (an Athlon
> >>XP2000) the levels load
> >>in about 16-20 seconds.
> >>
> >>
> >
> >There is a mistake in your reasoning, isn't it? You say that we shouldn't
look
> >at the size of the uncompressed XML data. Then you start profiling how
many
> >uncompressed MB/s can you load.
> >
> Well the engine IS loading uncompressed MB's. The decompression happens
> by another part
> of the loader so when I'm talking about load time performance I'm
> talking about the time to parse
> the UNCOMPRESSED file.
>
> >You should look at the number of compressed
> >MB/s. If running from harddisk, you should be loading between 5-20MB/s
> >(depending on speed of the drive). Anything less than 5MB, and you are
very
> >CPU limited. And you should _never_ be CPU limited while loading. Not
even
> >close. If you are CPU limited on a 2GHz CPU, think about all the poor
700MHz
> >Celeron users trying to load your game. There is no excuse for loading
data at
> >less speed than the media can stream.
> >
> Yes they are. You have to process the data. You have to send textures to
> the hardware (which is
> something that you cannot do in load time). Note that binary dump of a
> structure is not possible
> in our situation since Crystal Space is portable and has to run on
> various operating systems and
> CPU architectures. So we cannot depend on binary data being compatible.
> That means there is
> always SOME processing.
>
> Greetings,
>
> > Think about whether your users like to
> >wait 20 seconds to load a level. If it could be loaded in 5, or even
less...
> >
> >Can you get some more accurate profile?
> >
> >Alen
> >
> >
> >
> >-------------------------------------------------------
> >This SF.net email is sponsored by: SF.net Giveback Program.
> >Does SourceForge.net help you be more productive?  Does it
> >help you create better code?  SHARE THE LOVE, and help us help
> >YOU!  Click Here: http://sourceforge.net/donate/
> >_______________________________________________
> >Gamedevlists-general mailing list
> >Gam...@li...
> >https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> >Archives:
> >http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
> >
> >
> >
>

----- Original Message -----
From: "Alen Ladavac" <ale...@cr...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 8:43 PM
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library

> > The PlaneShift game (made with CS) has about 25 megabytes (uncompressed)
> > of XML files (aggregated
> > in four big files) of level data. On my Linux system at home (an Athlon
> > XP2000) the levels load
> > in about 16-20 seconds.
>
> There is a mistake in your reasoning, isn't it? You say that we shouldn't
look
> at the size of the uncompressed XML data. Then you start profiling how
many
> uncompressed MB/s can you load. You should look at the number of
compressed
> MB/s. If running from harddisk, you should be loading between 5-20MB/s
> (depending on speed of the drive). Anything less than 5MB, and you are
very
> CPU limited. And you should _never_ be CPU limited while loading. Not even
> close. If you are CPU limited on a 2GHz CPU, think about all the poor
700MHz
> Celeron users trying to load your game. There is no excuse for loading
data at
> less speed than the media can stream. Think about whether your users like
to
> wait 20 seconds to load a level. If it could be loaded in 5, or even
less...
>
> Can you get some more accurate profile?
>
> Alen
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.