gamedevlists-general — List for discussing general game development issues

Re: XML was RE: [GD-General] RE: A portable preferences library

[Jorrit T. <Jor...@uz...>]

Gareth Lewin wrote:

>
>There is no reason whatsoever for final proccessed data to be portable. You
>have your data in xml. You preprocess it into binary format for a specific
>build/platform. And you use that. Your load times go from 16seconds to 2 or
>3 seconds.
>
>And the amount of effort required to implement binary saving off of
>structures isn't anywhere near comparable with the effort required to tweak
>the speed of a graphics engine. Not to mention that 16-20 second load times
>would have me returning the game to the shop.
>  
>
Really? I'm amazed. What does 16 seconds mean for a game you're supposed 
to play for hours
(i.e. it is an MMORPG). That is negligable. If you really return a game 
to the shop for something
like that then I think you have weird priorities. No offense meant.

Greetings,

Re: XML was RE: [GD-General] RE: A portable preferences library

[J C L. <cl...@ka...>]

Without replying specifically to any point on this thread:

  http://www.faqs.org/docs/artu/ch05s01.html

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

[GD-General] (Really) XML for portable preferences library

[Jeff <je...@sp...>]

Me:
>> I'm surprised that no-one has said "use XML format please" - its
>> not that painful, given that you'll have a fairly limited schema
>> to implement.

Brian:
> So here's a question -- I know that whenever a discussion about text
> file formats comes up there is often a "use XML" grenade tossed into
> it.

And didn't it raise a firestorm this time, with an inordinate number of
people who didn't seem to be able to understand that a file format for a
PREFERENCES system need not have the same requirements as those for a 3D
LEVEL loader...

Brian:
>  Now, I can think of a lot of reasons NOT to use XML (i.e. it's
> overkill, for one), but I'm curious what real, tangible reasons for
> using XML exist, as opposed to a simple application specific structure
> (like INI files).

Me: (answering Brian's question)

A little has been said about the viewability of preference files in apps
like IE, and this is certainly an advantage.

XML also has standard(?) rules about encoding that let you deal with how
you'll store <> characters - your .INI needs similar rules for escape
characters (or quotes, or whatever...)

What I haven't seen in any of the discussion is anyone pointing out that XML
handles STRUCTURE very nicely.  At this point, your preferences library lets
the user write out integers and strings - however, its fairly
straight-forward to write out OBJECTS (or structs if you prefer).

Inhouse we use expat and have a simple shim wrapped around it that knows how
to write our favorite data structures, the most complex of which is a list
which contain lists (ad infinitum) which can contain ints, reals, strings,
points, lines, etc.  XML handles the nesting very nicely - if we used .INI
files we'd need to reinvent XML (or use some hideous key-naming scheme that
appended digits every time we went down a level)

Whilst you mightn't advocate points and lines for most peoples preferences,
think "saved window positions".  You might do:

<WINDOW name="toolpalette">
    <POS X=123 Y=456 />
    <SIZE H=100 W=200 />
</WINDOW>

As a bonus, if you use a more specific tag than POS, say WINDOWPOS, then
your XML loader (as opposed to your application) can even be smart enough to
clip its values based on the current screen size.  It means that you've
encoded more intelligence into the basic data types that you are storing,
this is true, but then you don't need to use them if you don't want to.

My main objection to XML, or perhaps its just expat's parser, is the
processing of whitespace. Expat calls you back each time it processes a
LINE, which means that any data types that get split over a line (perhaps by
someone's favorite XML editor) mean you need to implement your own buffering
to tie things back together.

Since the four lines of my email signature are a preference, its important
to me that any preference system handle multi-line strings correctly.

And finally, 

<MARKETTING>
If you use XML, you get to say "... and it uses industry-standard XML files"
and you get to watch the industry beat a path to your door because your
product is *obviously superior*.
</MARKETTING>

;-)

Jeff Laing <je...@sp...>
-------------------------------------------------------------------------------
"Meetings.  Don't we love meetings?  Every day.  Twice a day.  We talk." He got
on one elbow.  "I bet if I blew the conch this minute, they'd come running.
Then we'd be, you know, very solemn, and someone would say we ought to build a
jet, or a submarine or a TV set."       -- William Golding, "Lord of the Flies"

Re: XML was RE: [GD-General] RE: A portable preferences library but has nothing to do with them anymore

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 22:17:07 +0100 
Alen Ladavac <ale...@cr...> wrote:

> Can you please explain this calculation in more detail? You get 10MB
> /1.5s = 6.7M/s ? But you said you load 40MB in 11s, what I would say
> is 3.6 MB/s. Please note that you should not substract IO time from
> the total loading time, as CPU should be working in parallel with the
> DMA, otherwise you are doing something _very_ wrong.

The code in question does:

  sz = get_size_of_file ()
  read (h, buf, sz)
  parse (buf)

As such load and parse are entirely separate.  And yeah, by arithmetic
was off.  I'm parsing and cross linking (there's a fair few URI links in
the data) in roughly 6 seconds.

> Anyway, either 7MB/s, or 3.6 MB/s, this is just awfully slow. 

This is an old slow machine, a PII-333 -- I said it was far from
cutting-edge.  As it is a multi-user host there's also a fair bit going
on otherwise (mostly network IO rather than CPU or disk), but I didn't
check for to see what extent the other loads may have had.  I wasn't
trying for anything rigorous, merely a key to incite people to test with
their actual loadings, not a quick hack with a set of gene AFFY data.

> A 40xCDROM...

I was reading from disk possibly NFS over 100bT (can't check right now,
don't have access to that machine from where I am now).  That machine
doesn't have a CD.

> ...  has transfer rate of 6MB/s. You say you are parsing data at that
> rate? Those are certainly not some impressive numbers. :(

Certainly they're not impressive.  They weren't intended to be.  I
suggest you check them on a more representative host with code that
wasn't written as quickly as possible and for data that more closely
matches your loads.

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

Re: XML was RE: [GD-General] RE: A portable preferences library

[Alen L. <ale...@cr...>]

> Damn, I'm gonna fire that ranging scout!
:) 

>   double foo[large_number];
>   ...
>   read (h, foo, sizeof (foo[0]) * large_number) 
> 
> Is necessarily going to be cheaper than parsing an XML structure of
> large_number doubles.  The only question is whether the added expense is
> worth it on some level.  If you're shaving instructions...

Look, remember one thing: when doing things like this, always think about 
the bottleneck. Here, media speed is your bottleneck. Whatever you do, you 
can never be faster than it. Now, if you pipeline it well, you are free to 
use as much CPU time as media needs to stream it from disk. So the XML 
parser might as well have exact same speed as binary solution! 

The question is: is there some XML parser that can be fast enough to pull 
that trick, when run on some current CPU (and which), reading from some 
current media (and which)? Is there? 

And remember that it would need to be put in chain with an unzipper, for a 
fair race against a binary reader with unzipper, both running against the 
same file. 

But for example purposes, I'll accept an unzipper+XMLparser pair getting 
near media streaming speed, as a proof that XML doesn't suck. ;) 

Alen 

>> People started to argue that the verboseness of the format doesn't
>> hinder its performance. I say, ok, it might be possible, but show me
>> the numbers. Didn't see any yet. :/
> 
> Frankly and numbers you see need to be treated with a lot of salt, even
> the ones I post.  It is a simple enough thing to test with whatever
> parser you pick.   
> 
> -- 
> J C Lawrence
> ---------(*)                Satan, oscillate my metallic sonatas.
> cl...@ka...               He lived as a devil, eh?
> http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live. 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
 

Re: XML was RE: [GD-General] RE: A portable preferences library but has nothing to do with them anymore

[Alen L. <ale...@cr...>]

Can you please explain this calculation in more detail? 

You get 10MB /1.5s = 6.7M/s ? But you said you load 40MB in 11s, what I 
would say is 3.6 MB/s. Please note that you should not substract IO time 
from the total loading time, as CPU should be working in parallel with the 
DMA, otherwise you are doing something _very_ wrong. 

Anyway, either 7MB/s, or 3.6 MB/s, this is just awfully slow. A 40xCDROM has 
transfer rate of 6MB/s. You say you are parsing data at that rate? Those are 
certainly not some impressive numbers. :( 

Alen 

J C Lawrence writes: 

> On Tue, 16 Dec 2003 19:31:46 +0100 
> Pierre Terdiman <pie...@no...> wrote: 
> 
>> (Now I'm not saying that's the Right Way because, as you all should
>> know, there's no such thing as a Right Way in those recurrent A-vs-B
>> threads. When there's an actual Right Way, everybody agrees and people
>> don't waste their time discussing it).
> 
> <breaks down and cries tears of relief> 
> 
> While not game-related data (gene expression data as happens) I've found
> that I can load, parse, and cross link a 40Meg XML file with a
> moderately heavy rate of cross and external links (few score per K) in
> about 11 seconds on my far from cutting-edge desktop.  Raw IO time to
> simply suck the same data off disk is around 5 seconds.  That makes for
> a second and a half per 10Meg of XML data for parsing and linking, and
> this is without any attempt to optimise.   
> 
> -- 
> J C Lawrence
> ---------(*)                Satan, oscillate my metallic sonatas.
> cl...@ka...               He lived as a devil, eh?
> http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live. 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
 

Re: XML was RE: [GD-General] RE: A portable preferences library

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 21:53:36 +0100 
Alen Ladavac <ale...@cr...> wrote:

> Nope, you missed the boat. :) 

Damn, I'm gonna fire that ranging scout!

> The idea is that XML is a generic format, freeing you from writing
> your parser. 

Precisely.

> There is no much point storing plain blobs inside XML. Then, you can
> just dump it with fwrite() as well. 

Dumping a blob in XML really isn't that much more difficult.  You either
drop it inline as a base64 or as a MIME attachment in base64.  The data
size grows by 30% of course, but that's expected as a text
representation.

> But the question is: If you save your vertices in x="1" y="0"...etc
> format, can that still load as fast as if you just dump
> 3f8000000000000...etc? 

Of course not.  

  double foo[large_number];
  ...
  read (h, foo, sizeof (foo[0]) * large_number)

Is necessarily going to be cheaper than parsing an XML structure of
large_number doubles.  The only question is whether the added expense is
worth it on some level.  If you're shaving instructions...

> People started to argue that the verboseness of the format doesn't
> hinder its performance. I say, ok, it might be possible, but show me
> the numbers. Didn't see any yet. :/

Frankly and numbers you see need to be treated with a lot of salt, even
the ones I post.  It is a simple enough thing to test with whatever
parser you pick.  

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

Re: XML was RE: [GD-General] RE: A portable preferences library

[Alen L. <ale...@cr...>]

Nope, you missed the boat. :) The idea is that XML is a generic format, 
freeing you from writing your parser. There is no much point storing plain 
blobs inside XML. Then, you can just dump it with fwrite() as well. But the 
question is: If you save your vertices in x="1" y="0"...etc format, can that 
still load as fast as if you just dump 3f8000000000000...etc? (Hex dump from 
head - might have some errors ;) ). People started to argue that the 
verboseness of the format doesn't hinder its performance. I say, ok, it 
might be possible, but show me the numbers. Didn't see any yet. :/ 

Alen 


J C Lawrence writes: 

> On Tue, 16 Dec 2003 18:54:16 -0000 
> Alen Ladavac <ale...@cr...> wrote: 
> 
>> On a side note, this comment that "it is a HUGE ammount of work", is
>> based on a wrong assumption. Their problem is that the design was
>> wrong. If they went for generic binary format from the start, the work
>> would be near zero. But I digress.
> 
> No, you missed the boat. 
> 
> XML is an encoding format.  It says nothing about the data structures or
> data representation beyond the encoding.  More simply, through use of
> URI anchors and references there is little (nothing?) that can be
> represented in a serialised binary format that can't also be represented
> in XML.   
> 
> The discussion to date has on representing things like raw vertex data
> in a highly abstracted XML form.  That has obvious and unpleasant
> post-processing expenses.  If you don't want those costs, don't incur
> them: encode your already post-processed system-optimised yada yada data
> structures into XML.  There's nothing inherent to XML or binary data to
> prevent that.  You don't need to design or write your encoding format. 
> 
> XML is JUST an encoding format.  Well, it also happens to be a
> standardised encoding format, but that's more of a toolchain and code
> support issue.  At the end of the day where disk heads don't meet
> platters the effective differentials are in the storage and parsing
> overheads, not in the data structures.  The data that is represented is
> arbitrary, and for systems which don't need random reads (XML indexes
> badly) the performance differentials are invariably trivial. 
> 
> This written by someone who doesn't /like/ XML... 
> 
> -- 
> J C Lawrence
> ---------(*)                Satan, oscillate my metallic sonatas.
> cl...@ka...               He lived as a devil, eh?
> http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live. 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
 

Re: XML was RE: [GD-General] RE: A portable preferences library but has nothing to do with them anymore

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 19:31:46 +0100 
Pierre Terdiman <pie...@no...> wrote:

> (Now I'm not saying that's the Right Way because, as you all should
> know, there's no such thing as a Right Way in those recurrent A-vs-B
> threads. When there's an actual Right Way, everybody agrees and people
> don't waste their time discussing it).

<breaks down and cries tears of relief>

While not game-related data (gene expression data as happens) I've found
that I can load, parse, and cross link a 40Meg XML file with a
moderately heavy rate of cross and external links (few score per K) in
about 11 seconds on my far from cutting-edge desktop.  Raw IO time to
simply suck the same data off disk is around 5 seconds.  That makes for
a second and a half per 10Meg of XML data for parsing and linking, and
this is without any attempt to optimise.  

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

RE: XML was RE: [GD-General] RE: A portable preferences library b ut has nothing to do with them anymore

[Gareth L. <GL...@cl...>]

We are going in circles here, his data is zlib compressed, so the harddisk
costs are equivalent to a binary file (based on what Jorrit said).

And for the record, there are always many right ways. But there are also
wrong ways. 

Oh, and please Jorrit, I never meant to offend. I feel (based on offline
emails) that I might have offended. I'm sorry if that happened, as I think I
have posed only technical issues and nothing personal.

> -----Original Message-----
> From: Pierre Terdiman [mailto:pie...@no...]
> Sent: 16 December 2003 18:32
> To: gam...@li...
> Subject: Re: XML was RE: [GD-General] RE: A portable 
> preferences library
> but has nothing to do with them anymore
> 
> 
> > Hell, why not just pre-generate the kd-tree and store that 
> into the xml ?
> 
> Because the CPU is way faster than the HD, as already stated. 
> So, I wouldn't
> be surprised if generating the kd-tree was actually faster 
> than loading a
> huge text file (say XML). I certainly compute my AABB-trees 
> on-the-fly, for
> example.
> 
> (Now I'm not saying that's the Right Way because, as you all 
> should know,
> there's no such thing as a Right Way in those recurrent 
> A-vs-B threads. When
> there's an actual Right Way, everybody agrees and people 
> don't waste their
> time discussing it).
> 
> Pierre
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign 
> up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell 
> to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
> 

Re: XML was RE: [GD-General] RE: A portable preferences library but has nothing to do with them anymore

[Pierre T. <pie...@no...>]

> Hell, why not just pre-generate the kd-tree and store that into the xml ?

Because the CPU is way faster than the HD, as already stated. So, I wouldn't
be surprised if generating the kd-tree was actually faster than loading a
huge text file (say XML). I certainly compute my AABB-trees on-the-fly, for
example.

(Now I'm not saying that's the Right Way because, as you all should know,
there's no such thing as a Right Way in those recurrent A-vs-B threads. When
there's an actual Right Way, everybody agrees and people don't waste their
time discussing it).

Pierre

Re: XML was RE: [GD-General] RE: A portable preferences library

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 18:54:16 -0000 
Alen Ladavac <ale...@cr...> wrote:

> On a side note, this comment that "it is a HUGE ammount of work", is
> based on a wrong assumption. Their problem is that the design was
> wrong. If they went for generic binary format from the start, the work
> would be near zero. But I digress.

No, you missed the boat.

XML is an encoding format.  It says nothing about the data structures or
data representation beyond the encoding.  More simply, through use of
URI anchors and references there is little (nothing?) that can be
represented in a serialised binary format that can't also be represented
in XML.  

The discussion to date has on representing things like raw vertex data
in a highly abstracted XML form.  That has obvious and unpleasant
post-processing expenses.  If you don't want those costs, don't incur
them: encode your already post-processed system-optimised yada yada data
structures into XML.  There's nothing inherent to XML or binary data to
prevent that.  You don't need to design or write your encoding format.

XML is JUST an encoding format.  Well, it also happens to be a
standardised encoding format, but that's more of a toolchain and code
support issue.  At the end of the day where disk heads don't meet
platters the effective differentials are in the storage and parsing
overheads, not in the data structures.  The data that is represented is
arbitrary, and for systems which don't need random reads (XML indexes
badly) the performance differentials are invariably trivial.

This written by someone who doesn't /like/ XML...

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

RE: XML was RE: [GD-General] RE: A portable preferences library b ut has nothing to do with them anymore

[Gareth L. <GL...@cl...>]

But I think the point has been missed. The issue here (the reason I entered
this discussion) was that not only the data format was (imo) bad (text file
not parsed) but a lot of proccessing goes on with the data. He mentioned
kd-tree generation for example.

Hell, why not just pre-generate the kd-tree and store that into the xml ?
Surely that is a step in the right direction ?

> -----Original Message-----
> From: Alen Ladavac [mailto:ale...@cr...]
> Sent: 16 December 2003 18:54
> To: gam...@li...
> Subject: Re: XML was RE: [GD-General] RE: A portable 
> preferences library
> 
> 
> Gareth and Jorrit are going each too far in their own 
> direction. I believe
> that the reality lies in between. Putting everything into 
> platform-specific
> down-to-the-bit-optimized binary data is killing a fly with a 
> cannon, for most
> purposes. The only place where the load peformance matters so 
> much is if you
> want to stream your data during gameplay, and if you need it 
> quickly, so you
> have very little CPU to spend. Other than that, the only 
> thing that you should
> care about is that you are limited by HDD throughput, not by 
> CPU speed. As
> long as your CPU is ahead of the HDD, you are fine.
> 
> What I do is to have a binary format that is universal and 
> that can load any
> kind of objects. It can swap the bytes for endianness, etc. 
> but it knows in
> advance what layout looks like (in endianness neutral 
> format), and doesn't
> have to parse ASCII, let alone generic XML syntax. This is 
> not giving any
> generality away, while still allowing you to keep you loading 
> speed under
> control. I don't know of any platform where the CPU is so 
> slow that it cannot
> perform endianness swapping and such small adjustments while 
> loading. But
> going towards XML might be just a bit of an overkill. I say 
> _might_, not _is_.
> ... because I asked the original question about loading speed 
> plainly because
> I am interested in whether XML can be as efficient as a 
> binary format. I
> belive that it is not very likely, but not impossiible, and I 
> will accept any
> well profiled arguments. While this:
> 
> >In case of CS making such a faster loader
> >really is a HUGE amount of work. We have LOTS of different 
> data to save
> 
> is not something like that. I understand that CS team is not 
> working on a
> budget, and it is great that they do stuff for fun and for 
> others (though back
> when I was doing coding for my own fun, I wanted the results 
> to be perfect as
> well, but thats a different topic). But priorities of CS team 
> _have nothing to
> do_ with XML qualities. If I recall, it was the performance 
> of XML parsers
> that was discussed, not the idea of whether a loader should 
> be fast or not. If
> I say that my loader should be fast, then I guess it should. 
> I was just hoping
> that someone can show me some exact numbers.
> 
> On a side note, this comment that "it is a HUGE ammount of 
> work", is based on
> a wrong assumption. Their problem is that the design was 
> wrong. If they went
> for generic binary format from the start, the work would be 
> near zero. But I
> digress.
> 
> Can you Jorrit please get some more correct profiling data. 
> It doesn't matter
> that your app does preprocessing in-game, it is your choice. 
> But how fast is
> the parser itself?
> 
> Thanks,
> Alen
> 
> 
> 
> ----- Original Message -----
> From: "brian sharon" <pud...@po...>
> To: <gam...@li...>
> Sent: Tuesday, December 16, 2003 16:16
> Subject: Re: XML was RE: [GD-General] RE: A portable 
> preferences library
> 
> 
> > Would you please give it a rest?  In case you haven't 
> noticed there's
> > no "shop" involved for you to return his game to, as he's giving his
> > work away freely.  Jorrit is not doing this as a full-time job.  If
> > load times are not his first priority I can hardly blame 
> him, it's not
> > exactly the "fun" part of the job.
> >
> > Please visit http://crystal.sf.net to get clued in.
> >
> > --brian
> >
> > On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:
> >
> > > Sorry, but you are just wrong.
> > >
> > >>> Lets not ignore the fact that loading bits of the level is
> > >> way easier with a
> > >>> binary specific format that can be dumped directly into data
> > >> structures.
> > >>>
> > >>>
> > >> Yes but that's not portable. CS has to run on all kinds 
> of operating
> > >> systems and cpu's. We can't
> > >> dump binary structures like that. The endianess wouldn't be
> > >> correct and
> > >> padding can differ
> > >> too.
> > >>
> > >
> > > There is no reason whatsoever for final proccessed data to be
> > > portable. You
> > > have your data in xml. You preprocess it into binary format for a
> > > specific
> > > build/platform. And you use that. Your load times go from 
> 16seconds to
> > > 2 or
> > > 3 seconds.
> > >
> > > And the amount of effort required to implement binary 
> saving off of
> > > structures isn't anywhere near comparable with the effort 
> required to
> > > tweak
> > > the speed of a graphics engine. Not to mention that 16-20 
> second load
> > > times
> > > would have me returning the game to the shop.
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.net email is sponsored by: IBM Linux Tutorials.
> > > Become an expert in LINUX or just sharpen your skills.  
> Sign up for
> > > IBM's
> > > Free Linux Tutorials.  Learn everything from the bash shell to sys
> > > admin.
> > > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> > > _______________________________________________
> > > Gamedevlists-general mailing list
> > > Gam...@li...
> > > https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> > > Archives:
> > > http://sourceforge.net/mailarchive/forum.php?forum_id=557
> > >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IBM Linux Tutorials.
> > Become an expert in LINUX or just sharpen your skills.  
> Sign up for IBM's
> > Free Linux Tutorials.  Learn everything from the bash shell 
> to sys admin.
> > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> > _______________________________________________
> > Gamedevlists-general mailing list
> > Gam...@li...
> > https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> > Archives:
> > http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign 
> up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell 
> to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
> 

Re: XML was RE: [GD-General] RE: A portable preferences library

[brian s. <pud...@po...>]

Well for the record I make commercial games, and our engine does 
extensive platform-based preprocessing so that level loads are a single 
read into memory, and yes, it's the "right" way, and oh by the way 
we've drifted far, far away from portable preferences libraries.

My point was merely that load times might not be the #1 priority for 
someone who is doing this in his spare time, and it's probably more 
productive to note that and move on.

--brian

On Tuesday, December 16, 2003, at 09:31  AM, Gareth Lewin wrote:

> I'm sorry Brian, but that is besides the point. We were discussing how 
> best
> to do something, and his argument wasn't just one of time, but that 
> his idea
> was better.
>
> But yeah, I guess we should concentrate on the fun and not discuss the 
> right
> way to do stuff. Since like no-one on this list makes commercial games.
>
>> -----Original Message-----
>> From: brian sharon [mailto:pud...@po...]
>> Sent: 16 December 2003 16:16
>> To: gam...@li...
>> Subject: Re: XML was RE: [GD-General] RE: A portable
>> preferences library
>>
>>
>> Would you please give it a rest?  In case you haven't noticed there's
>> no "shop" involved for you to return his game to, as he's giving his
>> work away freely.  Jorrit is not doing this as a full-time job.  If
>> load times are not his first priority I can hardly blame him,
>> it's not
>> exactly the "fun" part of the job.
>>
>> Please visit http://crystal.sf.net to get clued in.
>>
>> --brian
>>
>> On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:
>>
>>> Sorry, but you are just wrong.
>>>
>>>>> Lets not ignore the fact that loading bits of the level is
>>>> way easier with a
>>>>> binary specific format that can be dumped directly into data
>>>> structures.
>>>>>
>>>>>
>>>> Yes but that's not portable. CS has to run on all kinds of
>> operating
>>>> systems and cpu's. We can't
>>>> dump binary structures like that. The endianess wouldn't be
>>>> correct and
>>>> padding can differ
>>>> too.
>>>>
>>>
>>> There is no reason whatsoever for final proccessed data to be
>>> portable. You
>>> have your data in xml. You preprocess it into binary format for a
>>> specific
>>> build/platform. And you use that. Your load times go from
>> 16seconds to
>>> 2 or
>>> 3 seconds.
>>>
>>> And the amount of effort required to implement binary saving off of
>>> structures isn't anywhere near comparable with the effort
>> required to
>>> tweak
>>> the speed of a graphics engine. Not to mention that 16-20
>> second load
>>> times
>>> would have me returning the game to the shop.

Re: XML was RE: [GD-General] RE: A portable preferences library

[Alen L. <ale...@cr...>]

Gareth and Jorrit are going each too far in their own direction. I believe
that the reality lies in between. Putting everything into platform-specific
down-to-the-bit-optimized binary data is killing a fly with a cannon, for most
purposes. The only place where the load peformance matters so much is if you
want to stream your data during gameplay, and if you need it quickly, so you
have very little CPU to spend. Other than that, the only thing that you should
care about is that you are limited by HDD throughput, not by CPU speed. As
long as your CPU is ahead of the HDD, you are fine.

What I do is to have a binary format that is universal and that can load any
kind of objects. It can swap the bytes for endianness, etc. but it knows in
advance what layout looks like (in endianness neutral format), and doesn't
have to parse ASCII, let alone generic XML syntax. This is not giving any
generality away, while still allowing you to keep you loading speed under
control. I don't know of any platform where the CPU is so slow that it cannot
perform endianness swapping and such small adjustments while loading. But
going towards XML might be just a bit of an overkill. I say _might_, not _is_.
... because I asked the original question about loading speed plainly because
I am interested in whether XML can be as efficient as a binary format. I
belive that it is not very likely, but not impossiible, and I will accept any
well profiled arguments. While this:

>In case of CS making such a faster loader
>really is a HUGE amount of work. We have LOTS of different data to save

is not something like that. I understand that CS team is not working on a
budget, and it is great that they do stuff for fun and for others (though back
when I was doing coding for my own fun, I wanted the results to be perfect as
well, but thats a different topic). But priorities of CS team _have nothing to
do_ with XML qualities. If I recall, it was the performance of XML parsers
that was discussed, not the idea of whether a loader should be fast or not. If
I say that my loader should be fast, then I guess it should. I was just hoping
that someone can show me some exact numbers.

On a side note, this comment that "it is a HUGE ammount of work", is based on
a wrong assumption. Their problem is that the design was wrong. If they went
for generic binary format from the start, the work would be near zero. But I
digress.

Can you Jorrit please get some more correct profiling data. It doesn't matter
that your app does preprocessing in-game, it is your choice. But how fast is
the parser itself?

Thanks,
Alen



----- Original Message -----
From: "brian sharon" <pud...@po...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 16:16
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library


> Would you please give it a rest?  In case you haven't noticed there's
> no "shop" involved for you to return his game to, as he's giving his
> work away freely.  Jorrit is not doing this as a full-time job.  If
> load times are not his first priority I can hardly blame him, it's not
> exactly the "fun" part of the job.
>
> Please visit http://crystal.sf.net to get clued in.
>
> --brian
>
> On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:
>
> > Sorry, but you are just wrong.
> >
> >>> Lets not ignore the fact that loading bits of the level is
> >> way easier with a
> >>> binary specific format that can be dumped directly into data
> >> structures.
> >>>
> >>>
> >> Yes but that's not portable. CS has to run on all kinds of operating
> >> systems and cpu's. We can't
> >> dump binary structures like that. The endianess wouldn't be
> >> correct and
> >> padding can differ
> >> too.
> >>
> >
> > There is no reason whatsoever for final proccessed data to be
> > portable. You
> > have your data in xml. You preprocess it into binary format for a
> > specific
> > build/platform. And you use that. Your load times go from 16seconds to
> > 2 or
> > 3 seconds.
> >
> > And the amount of effort required to implement binary saving off of
> > structures isn't anywhere near comparable with the effort required to
> > tweak
> > the speed of a graphics engine. Not to mention that 16-20 second load
> > times
> > would have me returning the game to the shop.
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IBM Linux Tutorials.
> > Become an expert in LINUX or just sharpen your skills.  Sign up for
> > IBM's
> > Free Linux Tutorials.  Learn everything from the bash shell to sys
> > admin.
> > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> > _______________________________________________
> > Gamedevlists-general mailing list
> > Gam...@li...
> > https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> > Archives:
> > http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
>

Re: XML was RE: [GD-General] RE: A portable preferences library

[Alen L. <ale...@cr...>]

> Well the engine IS loading uncompressed MB's. The decompression happens
> by another part
> of the loader so when I'm talking about load time performance I'm
> talking about the time to parse
> the UNCOMPRESSED file.

Please try to understand that I am not trying to bash you. I would like to
know the numbers. Those 16-20 seconds you gave is not a useable number,
because it is mixed with other preprocessing. And because unzipped file size
for XML is not very meaningless.

Hm. How can I explain this better? It doesn't matter how fast does it load in
terms of MB/s of some format that you use. What matters is how much useable
data per second can you load. Since we don't have any defined units for
"useable data", we can only compare that with some binary format. If you say
that XML size increase is neutralized by zipping, I would assume that you mean
that a zipped XML file would be of about same size as zipped binary file with
that same data in it. Now, the best way to judge the speed of XML parser would
then be to see how much _zipped_ MB/s can it load. If it is much below HDD
speed, then you cannot say that its costs are negligible.

Is that a more understandable explanation? And can you please give us that
info?

Thanks,
Alen



----- Original Message -----
From: "Jorrit Tyberghein" <Jor...@uz...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 14:50
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library


> Alen Ladavac wrote:
>
> >>The PlaneShift game (made with CS) has about 25 megabytes (uncompressed)
> >>of XML files (aggregated
> >>in four big files) of level data. On my Linux system at home (an Athlon
> >>XP2000) the levels load
> >>in about 16-20 seconds.
> >>
> >>
> >
> >There is a mistake in your reasoning, isn't it? You say that we shouldn't
look
> >at the size of the uncompressed XML data. Then you start profiling how many
> >uncompressed MB/s can you load.
> >
> Well the engine IS loading uncompressed MB's. The decompression happens
> by another part
> of the loader so when I'm talking about load time performance I'm
> talking about the time to parse
> the UNCOMPRESSED file.
>
> >You should look at the number of compressed
> >MB/s. If running from harddisk, you should be loading between 5-20MB/s
> >(depending on speed of the drive). Anything less than 5MB, and you are very
> >CPU limited. And you should _never_ be CPU limited while loading. Not even
> >close. If you are CPU limited on a 2GHz CPU, think about all the poor
700MHz
> >Celeron users trying to load your game. There is no excuse for loading data
at
> >less speed than the media can stream.
> >
> Yes they are. You have to process the data. You have to send textures to
> the hardware (which is
> something that you cannot do in load time). Note that binary dump of a
> structure is not possible
> in our situation since Crystal Space is portable and has to run on
> various operating systems and
> CPU architectures. So we cannot depend on binary data being compatible.
> That means there is
> always SOME processing.
>
> Greetings,
>
> > Think about whether your users like to
> >wait 20 seconds to load a level. If it could be loaded in 5, or even
less...
> >
> >Can you get some more accurate profile?
> >
> >Alen
> >
> >
> >
> >-------------------------------------------------------
> >This SF.net email is sponsored by: SF.net Giveback Program.
> >Does SourceForge.net help you be more productive?  Does it
> >help you create better code?  SHARE THE LOVE, and help us help
> >YOU!  Click Here: http://sourceforge.net/donate/
> >_______________________________________________
> >Gamedevlists-general mailing list
> >Gam...@li...
> >https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> >Archives:
> >http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
> >
> >
> >
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive?  Does it
> help you create better code?  SHARE THE LOVE, and help us help
> YOU!  Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
>

RE: XML was RE: [GD-General] RE: A portable preferences library

[Mat N. \(BUNGIE\) <mat...@mi...>]

That's a pretty arrogant way to look at this discussion. Small load
times improve every aspect of a project, commercial or not. It's faster
to debug, faster to load, faster to play, and in general creates the
impression of a much more efficient product.

MSN

-----Original Message-----
From: gam...@li...
[mailto:gam...@li...] On Behalf Of
Gareth Lewin
Sent: Tuesday, December 16, 2003 9:32 AM
To: gam...@li...
Subject: RE: XML was RE: [GD-General] RE: A portable preferences library

I'm sorry Brian, but that is besides the point. We were discussing how
best
to do something, and his argument wasn't just one of time, but that his
idea
was better.=20

But yeah, I guess we should concentrate on the fun and not discuss the
right
way to do stuff. Since like no-one on this list makes commercial games.

> -----Original Message-----
> From: brian sharon [mailto:pud...@po...]
> Sent: 16 December 2003 16:16
> To: gam...@li...
> Subject: Re: XML was RE: [GD-General] RE: A portable=20
> preferences library
>=20
>=20
> Would you please give it a rest?  In case you haven't noticed there's=20
> no "shop" involved for you to return his game to, as he's giving his=20
> work away freely.  Jorrit is not doing this as a full-time job.  If=20
> load times are not his first priority I can hardly blame him,=20
> it's not=20
> exactly the "fun" part of the job.
>=20
> Please visit http://crystal.sf.net to get clued in.
>=20
> --brian
>=20
> On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:
>=20
> > Sorry, but you are just wrong.
> >
> >>> Lets not ignore the fact that loading bits of the level is
> >> way easier with a
> >>> binary specific format that can be dumped directly into data
> >> structures.
> >>>
> >>>
> >> Yes but that's not portable. CS has to run on all kinds of=20
> operating
> >> systems and cpu's. We can't
> >> dump binary structures like that. The endianess wouldn't be
> >> correct and
> >> padding can differ
> >> too.
> >>
> >
> > There is no reason whatsoever for final proccessed data to be=20
> > portable. You
> > have your data in xml. You preprocess it into binary format for a=20
> > specific
> > build/platform. And you use that. Your load times go from=20
> 16seconds to=20
> > 2 or
> > 3 seconds.
> >
> > And the amount of effort required to implement binary saving off of
> > structures isn't anywhere near comparable with the effort=20
> required to=20
> > tweak
> > the speed of a graphics engine. Not to mention that 16-20=20
> second load=20
> > times
> > would have me returning the game to the shop.
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IBM Linux Tutorials.
> > Become an expert in LINUX or just sharpen your skills.  Sign up for=20
> > IBM's
> > Free Linux Tutorials.  Learn everything from the bash shell to sys=20
> > admin.
> > Click now! =
http://ads.osdn.com/?ad_id=3D1278&alloc_id=3D3371&op=3Dclick
> > _______________________________________________
> > Gamedevlists-general mailing list
> > Gam...@li...
> > https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> > Archives:
> > http://sourceforge.net/mailarchive/forum.php?forum_id=3D557
> >
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign=20
> up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell=20
> to sys admin.
> Click now! =
http://ads.osdn.com/?ad_id=3D1278&alloc_id=3D3371&op=3Dclick
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=3D557
>=20


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for
IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys
admin.
Click now! http://ads.osdn.com/?ad_id=3D1278&alloc_id=3D3371&op=3Dclick
_______________________________________________
Gamedevlists-general mailing list
Gam...@li...
https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_id=3D557

RE: XML was RE: [GD-General] RE: A portable preferences library

[Gareth L. <GL...@cl...>]

I'm sorry Brian, but that is besides the point. We were discussing how best
to do something, and his argument wasn't just one of time, but that his idea
was better. 

But yeah, I guess we should concentrate on the fun and not discuss the right
way to do stuff. Since like no-one on this list makes commercial games.

> -----Original Message-----
> From: brian sharon [mailto:pud...@po...]
> Sent: 16 December 2003 16:16
> To: gam...@li...
> Subject: Re: XML was RE: [GD-General] RE: A portable 
> preferences library
> 
> 
> Would you please give it a rest?  In case you haven't noticed there's 
> no "shop" involved for you to return his game to, as he's giving his 
> work away freely.  Jorrit is not doing this as a full-time job.  If 
> load times are not his first priority I can hardly blame him, 
> it's not 
> exactly the "fun" part of the job.
> 
> Please visit http://crystal.sf.net to get clued in.
> 
> --brian
> 
> On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:
> 
> > Sorry, but you are just wrong.
> >
> >>> Lets not ignore the fact that loading bits of the level is
> >> way easier with a
> >>> binary specific format that can be dumped directly into data
> >> structures.
> >>>
> >>>
> >> Yes but that's not portable. CS has to run on all kinds of 
> operating
> >> systems and cpu's. We can't
> >> dump binary structures like that. The endianess wouldn't be
> >> correct and
> >> padding can differ
> >> too.
> >>
> >
> > There is no reason whatsoever for final proccessed data to be 
> > portable. You
> > have your data in xml. You preprocess it into binary format for a 
> > specific
> > build/platform. And you use that. Your load times go from 
> 16seconds to 
> > 2 or
> > 3 seconds.
> >
> > And the amount of effort required to implement binary saving off of
> > structures isn't anywhere near comparable with the effort 
> required to 
> > tweak
> > the speed of a graphics engine. Not to mention that 16-20 
> second load 
> > times
> > would have me returning the game to the shop.
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IBM Linux Tutorials.
> > Become an expert in LINUX or just sharpen your skills.  Sign up for 
> > IBM's
> > Free Linux Tutorials.  Learn everything from the bash shell to sys 
> > admin.
> > Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> > _______________________________________________
> > Gamedevlists-general mailing list
> > Gam...@li...
> > https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> > Archives:
> > http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign 
> up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell 
> to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
> 

[GD-General] when and why we use XML

[<ns...@vs...>]

Brian Hook Wrote:-
/
/So here's a question -- I know that whenever a discussion about text 
/file formats comes up there is often a "use XML" grenade tossed into 
/it.  Now, I can think of a lot of reasons NOT to use XML (i.e. it's 
/overkill, for one), but I'm curious what real, tangible reasons for 
/using XML exist, as opposed to a simple application specific structure 
/(like INI files).

A few places where I have in the past found it fruitful to use XML:-

(1) Where a subset of a level has to be set up by a bunch of programmers and then "handed over" to a bunch of non-programmer level designer types: who need to then populate the data-set for a whole level OR multiple levels and then get back to a programming team with whether the feature set provided to them and under development has any glaring deficiencies. Oftentime in cases like this it helps to have the system architect/lead programmer XMLise the thingie - and pass it on for development of that and similar thingies to somewhat more junior members of the team who then have the onus of interacting directly with game designers - who then use XMLized tools to validate and populate data sets being prepared and tested by them.
For INI files there's a danger that members of the programming team may just pick up INI-file editor type pet projects OR that due to political reasons OR maturity(especially during the mid-lifecycle stages of a medium sized project) relatively members of the level design team may get into arguments OR raise red-flags and debates about the appropriateness of the feature set; and whether or not they need validate and feed the data set or not.

In situations like these which are common place (eg: (a) when setting up a few opponents with "individual opponent" properties and OR (b) When setting up data-sets section-wise varying level properties) using a somewhat more standardized format where it's easier to get input validating tools - it's useful to use XMLized data formats.

(2) When doing off-shore development - where sections of code are being worked on by a non-captive programming team. (reasons same as above; also, when there's a client-provider relationship; then people on the client's side are sometimes more likely to ask for more)

In both of the above cases, you could in principle use INI files editable thru text files; but experience has taught us that it's often not so, and feature demands are somewhat hard to ignore when one boasts of client satisfaction as one of the things that set oneself as a team apart from the rest.

This is not a general answer - but in our case - this is what experience has taught us abt where it makes sense - politically and commercially to use XML/XMLized data sets.

Hope that helps...

Regards,
Nalin

XML was RE: [GD-General] RE: A portable preferences library

[<ns...@vs...>]

> I like to clarify this a bit more. In case of CS making such a faster
loader
> really is a HUGE amount of work. We have LOTS of different data to save.
By
> using XML we can generalize the loader/saver a lot and this makes our
> project a lot
> more maintainable. Making specific optimized binary loaders and savers
> for all
> our different plugins is a huge undertaking and it also means a lot more
> code
> to maintain.
>
> Greetings,
>
Before anyone rides right in and starts taking shots at what Jorrit is
saying, I'd like to say that when data with crystal space is compressed into
ZIP files, the load times are actually pretty good.

I used and experimented a lot with CrystalSpace's source code some
months back (and I'm sure CS has improved considerably since then) and it
was pretty good - even load times wise - by any standards (I say that as
someone whose architected a commercial 3D engine technology; used Gamebryo;
used the
Quake engine; and experimented recompiling and changing those chunks of
Serious Sam's source code that're available).

Also, at a different time - years back; I experimented extensively with a
huffman coder coupled with a dictionary based compression scheme (basically
chunks of LHArc's source code that were freely available) and even on a 486,
the disk rather than the CPU was the bottleneck - and the computation time
to uncompress a 20 MB disk file (with 32 MB of RAM) was around 15 % of the
time it'd take to load the same data.
And with XML like data (label value pairs), it was often possible to get
compression ratios comparable to the same file in a binary format.
With XHTML documents one could often get extremely decent compression
ratios.

I would therefore imagine that if someone really were interested in using CS
as their underlying engine, it may be better to spend energy writing an
optimized ZIP file loader/decompressor (if the existing one's are not good
enough - I personally would imagine that either you could find existing
decompressors that're good enough/fast enough OR you can find something
that'd fit your purpose- maybe even a commercially available linkable
library).

I would therefore request all of us here who're in this discussion to add a
"PS: I use crystal space" or a "PS: I dont use crystal space" or "PS: I have
never even taken a shot at compiling the CS code" - and chances are that
most of us wouldnt have even touched CS with a bargepole.

Also, I feel that the lessons the people who're working on it learnt with
crystal space are extremely relevant - and it is counter productive to just
debate these points in a way that tries to look for revolutionary
discoveries or extreme stupidities of the "oooh... I found a mount everest
at the bottom of the ocean" or a "oh my God! that is horrible" rather than
looking at CS as being an evolutionary shared learning experience. That
defeats the whole point of both this group and of CS.

I hope that helps...

Best Regards,
Nalin
----- Original Message -----
From: "Jorrit Tyberghein" <Jor...@uz...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 8:20 PM
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library


> Alen Ladavac wrote:
>
> >>The PlaneShift game (made with CS) has about 25 megabytes (uncompressed)
> >>of XML files (aggregated
> >>in four big files) of level data. On my Linux system at home (an Athlon
> >>XP2000) the levels load
> >>in about 16-20 seconds.
> >>
> >>
> >
> >There is a mistake in your reasoning, isn't it? You say that we shouldn't
look
> >at the size of the uncompressed XML data. Then you start profiling how
many
> >uncompressed MB/s can you load.
> >
> Well the engine IS loading uncompressed MB's. The decompression happens
> by another part
> of the loader so when I'm talking about load time performance I'm
> talking about the time to parse
> the UNCOMPRESSED file.
>
> >You should look at the number of compressed
> >MB/s. If running from harddisk, you should be loading between 5-20MB/s
> >(depending on speed of the drive). Anything less than 5MB, and you are
very
> >CPU limited. And you should _never_ be CPU limited while loading. Not
even
> >close. If you are CPU limited on a 2GHz CPU, think about all the poor
700MHz
> >Celeron users trying to load your game. There is no excuse for loading
data at
> >less speed than the media can stream.
> >
> Yes they are. You have to process the data. You have to send textures to
> the hardware (which is
> something that you cannot do in load time). Note that binary dump of a
> structure is not possible
> in our situation since Crystal Space is portable and has to run on
> various operating systems and
> CPU architectures. So we cannot depend on binary data being compatible.
> That means there is
> always SOME processing.
>
> Greetings,
>
> > Think about whether your users like to
> >wait 20 seconds to load a level. If it could be loaded in 5, or even
less...
> >
> >Can you get some more accurate profile?
> >
> >Alen
> >
> >
> >
> >-------------------------------------------------------
> >This SF.net email is sponsored by: SF.net Giveback Program.
> >Does SourceForge.net help you be more productive?  Does it
> >help you create better code?  SHARE THE LOVE, and help us help
> >YOU!  Click Here: http://sourceforge.net/donate/
> >_______________________________________________
> >Gamedevlists-general mailing list
> >Gam...@li...
> >https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> >Archives:
> >http://sourceforge.net/mailarchive/forum.php?forum_id=557
> >
> >
> >
> >
>

----- Original Message -----
From: "Alen Ladavac" <ale...@cr...>
To: <gam...@li...>
Sent: Tuesday, December 16, 2003 8:43 PM
Subject: Re: XML was RE: [GD-General] RE: A portable preferences library


> > The PlaneShift game (made with CS) has about 25 megabytes (uncompressed)
> > of XML files (aggregated
> > in four big files) of level data. On my Linux system at home (an Athlon
> > XP2000) the levels load
> > in about 16-20 seconds.
>
> There is a mistake in your reasoning, isn't it? You say that we shouldn't
look
> at the size of the uncompressed XML data. Then you start profiling how
many
> uncompressed MB/s can you load. You should look at the number of
compressed
> MB/s. If running from harddisk, you should be loading between 5-20MB/s
> (depending on speed of the drive). Anything less than 5MB, and you are
very
> CPU limited. And you should _never_ be CPU limited while loading. Not even
> close. If you are CPU limited on a 2GHz CPU, think about all the poor
700MHz
> Celeron users trying to load your game. There is no excuse for loading
data at
> less speed than the media can stream. Think about whether your users like
to
> wait 20 seconds to load a level. If it could be loaded in 5, or even
less...
>
> Can you get some more accurate profile?
>
> Alen
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.

Re: XML was RE: [GD-General] RE: A portable preferences library

[Eero P. <epa...@ko...>]

Gareth Lewin wrote:

> How is that valid ? That's the same opengl/directx linux/windows type
> argument. In what way is it related or productive to the discussion ? I
> prefer msxml mainly because everyone with IE6 on their machine has it, less
> to download. And Xerces (admittedly I haven't tried it in 2 years) had an

There are of course different versions of  msxml, just installed one
software which required me to fetch an update from Microsoft.
(the software was actually clever enough to automatically download it,
except that it was not clever enough to do the download for
non English localized WindowsXP, and I think the required download
package is not that clearly named on Microsoft site...)


		Eero

Re: XML was RE: [GD-General] RE: A portable preferences library

[brian s. <pud...@po...>]

Would you please give it a rest?  In case you haven't noticed there's 
no "shop" involved for you to return his game to, as he's giving his 
work away freely.  Jorrit is not doing this as a full-time job.  If 
load times are not his first priority I can hardly blame him, it's not 
exactly the "fun" part of the job.

Please visit http://crystal.sf.net to get clued in.

--brian

On Tuesday, December 16, 2003, at 07:59  AM, Gareth Lewin wrote:

> Sorry, but you are just wrong.
>
>>> Lets not ignore the fact that loading bits of the level is
>> way easier with a
>>> binary specific format that can be dumped directly into data
>> structures.
>>>
>>>
>> Yes but that's not portable. CS has to run on all kinds of operating
>> systems and cpu's. We can't
>> dump binary structures like that. The endianess wouldn't be
>> correct and
>> padding can differ
>> too.
>>
>
> There is no reason whatsoever for final proccessed data to be 
> portable. You
> have your data in xml. You preprocess it into binary format for a 
> specific
> build/platform. And you use that. Your load times go from 16seconds to 
> 2 or
> 3 seconds.
>
> And the amount of effort required to implement binary saving off of
> structures isn't anywhere near comparable with the effort required to 
> tweak
> the speed of a graphics engine. Not to mention that 16-20 second load 
> times
> would have me returning the game to the shop.
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for 
> IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys 
> admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
>

RE: XML was RE: [GD-General] RE: A portable preferences library

[Gareth L. <GL...@cl...>]

> Oddly enough, there is no single right or wrong way of 
> looking at these 
> issues because there is no single right definition of game data.

There might be many right ways, but there are definitly some wrong ways.
Loading up vertex and face data, and then actually processing said data
(buidling kd-trees, bsd trees, light map calculatio, stripping etc) is just
wrong.

There might be some grey areas, but there is still a lot of stuff which is
obviously xml (config settings, keymaps etc) and some which is obviously
binary data.

RE: XML was RE: [GD-General] RE: A portable preferences library

[Gareth L. <GL...@cl...>]

Sorry, but you are just wrong.

> >Lets not ignore the fact that loading bits of the level is 
> way easier with a
> >binary specific format that can be dumped directly into data 
> structures.
> >  
> >
> Yes but that's not portable. CS has to run on all kinds of operating 
> systems and cpu's. We can't
> dump binary structures like that. The endianess wouldn't be 
> correct and 
> padding can differ
> too.
> 

There is no reason whatsoever for final proccessed data to be portable. You
have your data in xml. You preprocess it into binary format for a specific
build/platform. And you use that. Your load times go from 16seconds to 2 or
3 seconds.

And the amount of effort required to implement binary saving off of
structures isn't anywhere near comparable with the effort required to tweak
the speed of a graphics engine. Not to mention that 16-20 second load times
would have me returning the game to the shop.

Re: XML was RE: [GD-General] RE: A portable preferences library

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 15:13:52 +0200 
Ivan-Assen Ivanov <as...@ha...> wrote:

>> Technical reasons include that Xerces properly implements the open
>> standard in an opensource manner.

> This is a political reason, not a technical reason. 

It is both, mixed.

> I wouldn't base the choice of including a bloated mess of dozens of
> source files over a couple of lines of COM calls, on what is a proper
> opensource manner, or what habits MS has. It's a matter of
> engineering, not Slashdot karma.

As he stated, Xerces also implements the current draft to a degree that
is auditable and published, and is therefore largely without surprises.
This is a technical benefit, both in the certainty of what is supported
and how, and the adherence to standards and thus interop with other
adherent tools.

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.