Thread: Re: [Algorithms] In-place loaded data structures. (Page 3)
Brought to you by:
vexxed72
From: Alen L. <ale...@cr...> - 2005-11-30 13:16:53
|
> Absolutely, I understood that. That still means that the data is not > stored in the file as a blob, you just know that you can load it all as if > it were a blob if the endianness and size criteria are met. I'm simply > pointing out that you typically _can't_ store a vertex array as a blob > because of the endianness issues. Yeah, they are not stored as blobs, they are only loaded that way - sometimes. Overall, we agree on all points I believe. It is just a matter of "tuning in" the terminology. :) Cheers, Alen |
From: Jamie F. <ja...@qu...> - 2005-11-30 13:18:54
|
Alen Ladavac wrote: >> Absolutely, I understood that. That still means that the data is not >> stored in the file as a blob, you just know that you can load it all >> as if it were a blob if the endianness and size criteria are met. I'm >> simply pointing out that you typically _can't_ store a vertex array as >> a blob because of the endianness issues. > > > Yeah, they are not stored as blobs, they are only loaded that way - > sometimes. > > Overall, we agree on all points I believe. It is just a matter of > "tuning in" the terminology. :) Yup, terminology is definitely far from settled in this area :) Jamie |
From: Tom F. <tom...@ee...> - 2005-11-30 17:33:35
|
One trick is to store all the data and then all the metadata, and have them in two discrete blocks. Then you can load the whole thing, fix it up (version-convert, marshall, fix pointers, etc), and then discard the metadata. That way it doesn't hang around cluttering up memory. TomF. > -----Original Message----- > From: gda...@li... > [mailto:gda...@li...] On > Behalf Of Alen Ladavac > Sent: 30 November 2005 00:57 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. > > > >> Hm, I'm not sure why are you having any problems with > this? We're using > >> metadata for almost all content, from small config files > to meshes, > >> textures, animations and levels. (Sounds and video files > are standard > >> formats - wav, ogg....). Saved file overhead scales with > the number of > >> different types and smallest files are <1kb. > > > > Well, smallest size for us means including the metadata for all the > > objects. Sure, you could trim some out for limited > applications, but in > > practice for any meaningful application you'd be using > almost all the data > > types. > > Yes, that's what I said. The smallest file I was able to > found is 960 bytes, > together with all metadata for all objects contained in that > file. That's > just a set of parameters for a weapon. > > Perhaps we are talking about a slightly different thing. We > do not store > metadata for each object in the file, but only once for each > _datatype_ that > has an object in this file. So e.g. if you have a model that > has materials > in it, the CMaterial datatype is stored only once. All this > info is stored > in the file's header, and the rest of the file is binary data > just as if you > were using some fixed-format data file. It's the header that > describes the > objects. > > > Even with that tight binding, there's still room to deal > with endianness, > > but not class member changes. > > With the above types-in-the-header approach, there is. > > > >> For things that have large blobs of data (vertex arrays or > textures), the > >> system automatically detects when loading an array of > PODTs and just > >> slaps the raw data directly into memory. > > > > Absolutely; it's all about having the appropriate primitive > data types > > supported in your loading code, and big blob of data has to > be one of them > > :) > > Actually, you don't need to have specific BLOB datatypes. Any > "simple type" > (int, float, enum...) is considered "raw" if the files > endianness matches > that of the machine that loads the file. Then the "raw" > property propagates > up the type hierarchy and you get things like vertex arrays > automatically > detected as blobs. This is all done at load time, but is pretty low > overhead. > > HTH, > Alen > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=6188 > |
From: Alen L. <ale...@cr...> - 2005-11-30 22:14:00
|
> One trick is to store all the data and then all the metadata, and have them > in two discrete blocks. Then you can load the whole thing, fix it up > (version-convert, marshall, fix pointers, etc), and then discard the > metadata. That way it doesn't hang around cluttering up memory. One thing here is that you will assume that the data will just get loaded, used by only reading and will them get freed out in one heap block. You are not able do this with living data, right? Because if you do it that way you'd have to have heap block info saved together with it, if each object is a separate heap block. (Is this explanation apprehendable?) E.g. if you load a level that way, and entities within the level can be deleted/created during the game, you need each entity in its own memory block. Are you using this system only for e.g. models, or are you really leaving space for heap blocks' info within that data? Cheers, Alen |
From: Tom F. <tom...@ee...> - 2005-12-03 21:54:32
|
Living data generally (a) doesn't need format conversion (because it's short-lived, e.g. a savegame) and therefore (b) has a fixed data = definition. So you don't actually need to store the metadata in the file at all, = it's implicit in the program you're using. But serialising live data out to disk takes a lot of time anyway, = because you have to do a "gather" operation, i.e. grab a bunch of data from all = over memory and put it into one contiguous block to be saved. Unless you're just talking about changing values within the data, not changing its structure, which is trivial to do in any system :-) So yes, the answer is - this is mainly used for read-only data. There's = no reason you can't use it for read/write data (and obviously your = authoring pipeline will do read-modify-write operations all the time), but it's = not critical that saving stuff out is a high-speed operation - you're not = going to be streaming data _out_ during a game! Hmm... except multiplayer/MMO comms - you stream that out at runtime, = and it needs format conversion (usually endianness mashalling). But that's a = very different kettle of fish! TomF. > -----Original Message----- > From: gda...@li...=20 > [mailto:gda...@li...] On=20 > Behalf Of Alen Ladavac > Sent: 30 November 2005 14:14 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. >=20 >=20 > > One trick is to store all the data and then all the=20 > metadata, and have > them > > in two discrete blocks. Then you can load the whole thing, fix it up > > (version-convert, marshall, fix pointers, etc), and then discard the > > metadata. That way it doesn't hang around cluttering up memory. >=20 > One thing here is that you will assume that the data will=20 > just get loaded, > used by only reading and will them get freed out in one heap=20 > block. You are > not able do this with living data, right? Because if you do=20 > it that way > you'd have to have heap block info saved together with it, if=20 > each object is > a separate heap block. (Is this explanation apprehendable?)=20 > E.g. if you load > a level that way, and entities within the level can be deleted/created > during the game, you need each entity in its own memory=20 > block. Are you using > this system only for e.g. models, or are you really leaving=20 > space for heap > blocks' info within that data? >=20 > Cheers, > Alen >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep=20 > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. =20 > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 >=20 |
From: Alen L. <ale...@cr...> - 2005-12-04 07:38:12
|
That's not exactly what I was talking about. I ment this: Your level has entities in it. Those entities are mainly placed by level designers, but may also be changed, created and deleted during the game. It makes much sense to have each entity as a separate memory block (malloc-ed, or new-ed), so they can be created/deleted. In that case, how do you load all the data in one chunk of memory? Do you include heap's block headers in there and relocate them after load? Alen ----- Original Message ----- From: "Tom Forsyth" <tom...@ee...> To: <gda...@li...> Sent: Saturday, December 03, 2005 10:54 PM Subject: RE: [Algorithms] In-place loaded data structures. Living data generally (a) doesn't need format conversion (because it's short-lived, e.g. a savegame) and therefore (b) has a fixed data definition. So you don't actually need to store the metadata in the file at all, it's implicit in the program you're using. But serialising live data out to disk takes a lot of time anyway, because you have to do a "gather" operation, i.e. grab a bunch of data from all over memory and put it into one contiguous block to be saved. Unless you're just talking about changing values within the data, not changing its structure, which is trivial to do in any system :-) So yes, the answer is - this is mainly used for read-only data. There's no reason you can't use it for read/write data (and obviously your authoring pipeline will do read-modify-write operations all the time), but it's not critical that saving stuff out is a high-speed operation - you're not going to be streaming data _out_ during a game! Hmm... except multiplayer/MMO comms - you stream that out at runtime, and it needs format conversion (usually endianness mashalling). But that's a very different kettle of fish! TomF. > -----Original Message----- > From: gda...@li... > [mailto:gda...@li...] On > Behalf Of Alen Ladavac > Sent: 30 November 2005 14:14 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. > > > > One trick is to store all the data and then all the > metadata, and have > them > > in two discrete blocks. Then you can load the whole thing, fix it up > > (version-convert, marshall, fix pointers, etc), and then discard the > > metadata. That way it doesn't hang around cluttering up memory. > > One thing here is that you will assume that the data will > just get loaded, > used by only reading and will them get freed out in one heap > block. You are > not able do this with living data, right? Because if you do > it that way > you'd have to have heap block info saved together with it, if > each object is > a separate heap block. (Is this explanation apprehendable?) > E.g. if you load > a level that way, and entities within the level can be deleted/created > during the game, you need each entity in its own memory > block. Are you using > this system only for e.g. models, or are you really leaving > space for heap > blocks' info within that data? > > Cheers, > Alen > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=6188 > ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=ick _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_ida88 |
From: Tom F. <tom...@ee...> - 2005-12-04 08:02:49
|
> In that case, how do you load all the=20 > data in one chunk of memory? I don't think you do :-) Entities are dynamic structures, with hooks to all sorts of parts of = your game engine. I'm not sure it's even possible to simply load them off = disk - your pointer fixup would be incredibly complex, as you say. Far better would be to load an "initial state" structure (or more = likely, a whole list of them for all the entities in the level/next chunk/local region) and create the actual dynamic data from that structure. Then = throw the structure away. Actually using the same structure for live data as = the one you loaded off disk seems unwise. For one thing, the live one will probably be a lot bigger than the initial state structure you load off = disk (which may only be a hundred bytes or so) - no point waiting for all = that data to load, then having to do some complex fixup procedure. However, I know some people do indeed do this, simply by "playing" the = game until the start of the level, then dumping the whole of memory out to a = disk image. Then to load the level, just load that whole chunk of data back = in. This never really made much sense to me - you're waiting for the DVD/HDD = to load this big chunk of memory with sparse and complex data, and = meanwhile your powerful CPU is sitting nearly idle. It is simple though! And for streaming worlds (i.e. not level-based), it seems impractical at best. TomF. > -----Original Message----- > From: gda...@li...=20 > [mailto:gda...@li...] On=20 > Behalf Of Alen Ladavac > Sent: 03 December 2005 23:38 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. >=20 >=20 > That's not exactly what I was talking about. I ment this:=20 > Your level has > entities in it. Those entities are mainly placed by level=20 > designers, but may > also be changed, created and deleted during the game. It=20 > makes much sense to > have each entity as a separate memory block (malloc-ed, or=20 > new-ed), so they > can be created/deleted. In that case, how do you load all the=20 > data in one > chunk of memory? Do you include heap's block headers in there=20 > and relocate > them after load? >=20 > Alen >=20 >=20 > ----- Original Message ----- > From: "Tom Forsyth" <tom...@ee...> > To: <gda...@li...> > Sent: Saturday, December 03, 2005 10:54 PM > Subject: RE: [Algorithms] In-place loaded data structures. >=20 >=20 > Living data generally (a) doesn't need format conversion (because it's > short-lived, e.g. a savegame) and therefore (b) has a fixed=20 > data definition. > So you don't actually need to store the metadata in the file=20 > at all, it's > implicit in the program you're using. >=20 > But serialising live data out to disk takes a lot of time=20 > anyway, because > you have to do a "gather" operation, i.e. grab a bunch of=20 > data from all over > memory and put it into one contiguous block to be saved. >=20 > Unless you're just talking about changing values within the data, not > changing its structure, which is trivial to do in any system :-) >=20 > So yes, the answer is - this is mainly used for read-only=20 > data. There's no > reason you can't use it for read/write data (and obviously=20 > your authoring > pipeline will do read-modify-write operations all the time),=20 > but it's not > critical that saving stuff out is a high-speed operation -=20 > you're not going > to be streaming data _out_ during a game! >=20 > Hmm... except multiplayer/MMO comms - you stream that out at=20 > runtime, and it > needs format conversion (usually endianness mashalling). But=20 > that's a very > different kettle of fish! >=20 > TomF. >=20 > > -----Original Message----- > > From: gda...@li... > > [mailto:gda...@li...] On > > Behalf Of Alen Ladavac > > Sent: 30 November 2005 14:14 > > To: gda...@li... > > Subject: Re: [Algorithms] In-place loaded data structures. > > > > > > > One trick is to store all the data and then all the > > metadata, and have > > them > > > in two discrete blocks. Then you can load the whole=20 > thing, fix it up > > > (version-convert, marshall, fix pointers, etc), and then=20 > discard the > > > metadata. That way it doesn't hang around cluttering up memory. > > > > One thing here is that you will assume that the data will > > just get loaded, > > used by only reading and will them get freed out in one heap > > block. You are > > not able do this with living data, right? Because if you do > > it that way > > you'd have to have heap block info saved together with it, if > > each object is > > a separate heap block. (Is this explanation apprehendable?) > > E.g. if you load > > a level that way, and entities within the level can be=20 > deleted/created > > during the game, you need each entity in its own memory > > block. Are you using > > this system only for e.g. models, or are you really leaving > > space for heap > > blocks' info within that data? > > > > Cheers, > > Alen > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep > > through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. > > DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > > _______________________________________________ > > GDAlgorithms-list mailing list > > GDA...@li... > > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > > Archives: > > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 > > >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep=20 > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. =20 > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id=16865&op=3Dick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_ida88 >=20 >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep=20 > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. =20 > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 >=20 |
From: Alen L. <ale...@cr...> - 2005-12-04 10:13:32
|
Exactly what I'm saying. In presence of a "powerful-enough" CPU, it doesn't make any sense to concentrate on optimizing the amount of "data massaging" and trying to push all data into one large chunk. It is more useful to concentrate on minimizing seeks and amount of data that needs to be loaded. In such case, just pushing everything through a general-purpose metadata system gives you more oportunity for general optimizations, rather than trying to push everything to in-place structures. This assumes that the general purpose system is not just xml, but something more space-savy. This probably doesn't work for handhelds today, but for PC and nextgen it might be the simplest solution. Alen ----- Original Message ----- From: "Tom Forsyth" <tom...@ee...> To: <gda...@li...> Sent: Sunday, December 04, 2005 9:02 AM Subject: RE: [Algorithms] In-place loaded data structures. > In that case, how do you load all the > data in one chunk of memory? I don't think you do :-) Entities are dynamic structures, with hooks to all sorts of parts of your game engine. I'm not sure it's even possible to simply load them off disk - your pointer fixup would be incredibly complex, as you say. Far better would be to load an "initial state" structure (or more likely, a whole list of them for all the entities in the level/next chunk/local region) and create the actual dynamic data from that structure. Then throw the structure away. Actually using the same structure for live data as the one you loaded off disk seems unwise. For one thing, the live one will probably be a lot bigger than the initial state structure you load off disk (which may only be a hundred bytes or so) - no point waiting for all that data to load, then having to do some complex fixup procedure. However, I know some people do indeed do this, simply by "playing" the game until the start of the level, then dumping the whole of memory out to a disk image. Then to load the level, just load that whole chunk of data back in. This never really made much sense to me - you're waiting for the DVD/HDD to load this big chunk of memory with sparse and complex data, and meanwhile your powerful CPU is sitting nearly idle. It is simple though! And for streaming worlds (i.e. not level-based), it seems impractical at best. TomF. > -----Original Message----- > From: gda...@li... > [mailto:gda...@li...] On > Behalf Of Alen Ladavac > Sent: 03 December 2005 23:38 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. > > > That's not exactly what I was talking about. I ment this: > Your level has > entities in it. Those entities are mainly placed by level > designers, but may > also be changed, created and deleted during the game. It > makes much sense to > have each entity as a separate memory block (malloc-ed, or > new-ed), so they > can be created/deleted. In that case, how do you load all the > data in one > chunk of memory? Do you include heap's block headers in there > and relocate > them after load? > > Alen > > > ----- Original Message ----- > From: "Tom Forsyth" <tom...@ee...> > To: <gda...@li...> > Sent: Saturday, December 03, 2005 10:54 PM > Subject: RE: [Algorithms] In-place loaded data structures. > > > Living data generally (a) doesn't need format conversion (because it's > short-lived, e.g. a savegame) and therefore (b) has a fixed > data definition. > So you don't actually need to store the metadata in the file > at all, it's > implicit in the program you're using. > > But serialising live data out to disk takes a lot of time > anyway, because > you have to do a "gather" operation, i.e. grab a bunch of > data from all over > memory and put it into one contiguous block to be saved. > > Unless you're just talking about changing values within the data, not > changing its structure, which is trivial to do in any system :-) > > So yes, the answer is - this is mainly used for read-only > data. There's no > reason you can't use it for read/write data (and obviously > your authoring > pipeline will do read-modify-write operations all the time), > but it's not > critical that saving stuff out is a high-speed operation - > you're not going > to be streaming data _out_ during a game! > > Hmm... except multiplayer/MMO comms - you stream that out at > runtime, and it > needs format conversion (usually endianness mashalling). But > that's a very > different kettle of fish! > > TomF. > > > -----Original Message----- > > From: gda...@li... > > [mailto:gda...@li...] On > > Behalf Of Alen Ladavac > > Sent: 30 November 2005 14:14 > > To: gda...@li... > > Subject: Re: [Algorithms] In-place loaded data structures. > > > > > > > One trick is to store all the data and then all the > > metadata, and have > > them > > > in two discrete blocks. Then you can load the whole > thing, fix it up > > > (version-convert, marshall, fix pointers, etc), and then > discard the > > > metadata. That way it doesn't hang around cluttering up memory. > > > > One thing here is that you will assume that the data will > > just get loaded, > > used by only reading and will them get freed out in one heap > > block. You are > > not able do this with living data, right? Because if you do > > it that way > > you'd have to have heap block info saved together with it, if > > each object is > > a separate heap block. (Is this explanation apprehendable?) > > E.g. if you load > > a level that way, and entities within the level can be > deleted/created > > during the game, you need each entity in its own memory > > block. Are you using > > this system only for e.g. models, or are you really leaving > > space for heap > > blocks' info within that data? > > > > Cheers, > > Alen > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep > > through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. > > DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > GDAlgorithms-list mailing list > > GDA...@li... > > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > > Archives: > > http://sourceforge.net/mailarchive/forum.php?forum_id=6188 > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&op=ick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_ida88 > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=6188 > ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=ick _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_ida88 |
From: Tom F. <tom...@ee...> - 2005-12-04 21:07:46
|
Ah, see that's taking it too far IMHO. The majority of streamed assets = are static, read-only, and only have internal references (textures, meshes, collision hulls, etc). It is possible to load those as single chunks = with minor fixups, and as they're the majority of your data, the time saving = is significant even on high-end CPUs. In fact the main cost for processing = data is the memory bandwidth and latency (not actual processing speed as = such), and neither of those are going up very quickly. > In such case, just pushing everything through a=20 > general-purpose metadata > system gives you more oportunity for general optimizations,=20 > rather than > trying to push everything to in-place structures. This=20 The real trick is to make sure your in-place data structure IS a general-purpose metadata system. Then you get the best of both worlds. = The downside is code complication of course (or you could just buy it from someone who's already done the hard work :-) My point with the dynamic entities is that storing the in-memory entity itself to disk (in any format) is silly - it's hard work, and most of = that data is mutable or temporary. Rather, you want to store a description of that entitity that only has the details you care about. TomF. > -----Original Message----- > From: gda...@li...=20 > [mailto:gda...@li...] On=20 > Behalf Of Alen Ladavac > Sent: 04 December 2005 02:13 > To: gda...@li... > Subject: Re: [Algorithms] In-place loaded data structures. >=20 >=20 > Exactly what I'm saying. In presence of a "powerful-enough"=20 > CPU, it doesn't > make any sense to concentrate on optimizing the amount of=20 > "data massaging" > and trying to push all data into one large chunk. It is more useful to > concentrate on minimizing seeks and amount of data that needs=20 > to be loaded. > In such case, just pushing everything through a=20 > general-purpose metadata > system gives you more oportunity for general optimizations,=20 > rather than > trying to push everything to in-place structures. This=20 > assumes that the > general purpose system is not just xml, but something more=20 > space-savy. This > probably doesn't work for handhelds today, but for PC and=20 > nextgen it might > be the simplest solution. >=20 > Alen >=20 > ----- Original Message ----- > From: "Tom Forsyth" <tom...@ee...> > To: <gda...@li...> > Sent: Sunday, December 04, 2005 9:02 AM > Subject: RE: [Algorithms] In-place loaded data structures. >=20 >=20 > > In that case, how do you load all the > > data in one chunk of memory? >=20 > I don't think you do :-) >=20 > Entities are dynamic structures, with hooks to all sorts of=20 > parts of your > game engine. I'm not sure it's even possible to simply load=20 > them off disk - > your pointer fixup would be incredibly complex, as you say. >=20 > Far better would be to load an "initial state" structure (or=20 > more likely, a > whole list of them for all the entities in the level/next chunk/local > region) and create the actual dynamic data from that=20 > structure. Then throw > the structure away. Actually using the same structure for=20 > live data as the > one you loaded off disk seems unwise. For one thing, the live one will > probably be a lot bigger than the initial state structure you=20 > load off disk > (which may only be a hundred bytes or so) - no point waiting=20 > for all that > data to load, then having to do some complex fixup procedure. >=20 > However, I know some people do indeed do this, simply by=20 > "playing" the game > until the start of the level, then dumping the whole of=20 > memory out to a disk > image. Then to load the level, just load that whole chunk of=20 > data back in. > This never really made much sense to me - you're waiting for=20 > the DVD/HDD to > load this big chunk of memory with sparse and complex data,=20 > and meanwhile > your powerful CPU is sitting nearly idle. It is simple though! And for > streaming worlds (i.e. not level-based), it seems impractical at best. >=20 > TomF. >=20 > > -----Original Message----- > > From: gda...@li... > > [mailto:gda...@li...] On > > Behalf Of Alen Ladavac > > Sent: 03 December 2005 23:38 > > To: gda...@li... > > Subject: Re: [Algorithms] In-place loaded data structures. > > > > > > That's not exactly what I was talking about. I ment this: > > Your level has > > entities in it. Those entities are mainly placed by level > > designers, but may > > also be changed, created and deleted during the game. It > > makes much sense to > > have each entity as a separate memory block (malloc-ed, or > > new-ed), so they > > can be created/deleted. In that case, how do you load all the > > data in one > > chunk of memory? Do you include heap's block headers in there > > and relocate > > them after load? > > > > Alen > > > > > > ----- Original Message ----- > > From: "Tom Forsyth" <tom...@ee...> > > To: <gda...@li...> > > Sent: Saturday, December 03, 2005 10:54 PM > > Subject: RE: [Algorithms] In-place loaded data structures. > > > > > > Living data generally (a) doesn't need format conversion=20 > (because it's > > short-lived, e.g. a savegame) and therefore (b) has a fixed > > data definition. > > So you don't actually need to store the metadata in the file > > at all, it's > > implicit in the program you're using. > > > > But serialising live data out to disk takes a lot of time > > anyway, because > > you have to do a "gather" operation, i.e. grab a bunch of > > data from all over > > memory and put it into one contiguous block to be saved. > > > > Unless you're just talking about changing values within the=20 > data, not > > changing its structure, which is trivial to do in any system :-) > > > > So yes, the answer is - this is mainly used for read-only > > data. There's no > > reason you can't use it for read/write data (and obviously > > your authoring > > pipeline will do read-modify-write operations all the time), > > but it's not > > critical that saving stuff out is a high-speed operation - > > you're not going > > to be streaming data _out_ during a game! > > > > Hmm... except multiplayer/MMO comms - you stream that out at > > runtime, and it > > needs format conversion (usually endianness mashalling). But > > that's a very > > different kettle of fish! > > > > TomF. > > > > > -----Original Message----- > > > From: gda...@li... > > > [mailto:gda...@li...] On > > > Behalf Of Alen Ladavac > > > Sent: 30 November 2005 14:14 > > > To: gda...@li... > > > Subject: Re: [Algorithms] In-place loaded data structures. > > > > > > > > > > One trick is to store all the data and then all the > > > metadata, and have > > > them > > > > in two discrete blocks. Then you can load the whole > > thing, fix it up > > > > (version-convert, marshall, fix pointers, etc), and then > > discard the > > > > metadata. That way it doesn't hang around cluttering up memory. > > > > > > One thing here is that you will assume that the data will > > > just get loaded, > > > used by only reading and will them get freed out in one heap > > > block. You are > > > not able do this with living data, right? Because if you do > > > it that way > > > you'd have to have heap block info saved together with it, if > > > each object is > > > a separate heap block. (Is this explanation apprehendable?) > > > E.g. if you load > > > a level that way, and entities within the level can be > > deleted/created > > > during the game, you need each entity in its own memory > > > block. Are you using > > > this system only for e.g. models, or are you really leaving > > > space for heap > > > blocks' info within that data? > > > > > > Cheers, > > > Alen > > > > > > > > > > > > ------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. Do you grep > > > through log files > > > for problems? Stop! Download the new AJAX search engine=20 > that makes > > > searching your log files as easy as surfing the web. > > > DOWNLOAD SPLUNK! > > > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > > > _______________________________________________ > > > GDAlgorithms-list mailing list > > > GDA...@li... > > > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > > > Archives: > > > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 > > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep > > through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. > > DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_idv37&alloc_id=16865&op=3Dick > > _______________________________________________ > > GDAlgorithms-list mailing list > > GDA...@li... > > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > > Archives: > > http://sourceforge.net/mailarchive/forum.php?forum_ida88 > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep > > through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. > > DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > > _______________________________________________ > > GDAlgorithms-list mailing list > > GDA...@li... > > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > > Archives: > > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 > > >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep=20 > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. =20 > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id=16865&op=3Dick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_ida88 >=20 >=20 >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep=20 > through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. =20 > DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_id=3D6188 >=20 |
From: Alen L. <ale...@cr...> - 2005-12-05 09:37:33
|
>The real trick is to make sure your in-place data structure IS a >general-purpose metadata system. Then you get the best of both worlds. Hm? How could a general-purpose system be in-place? >downside is code complication of course (or you could just buy it from >someone who's already done the hard work :-) Do you have any examples in mind? >My point with the dynamic entities is that storing the in-memory entity >itself to disk (in any format) is silly - it's hard work, and most of that >data is mutable or temporary. Rather, you want to store a description of >that entitity that only has the details you care about. Exactly: A good general-purpose metadata system will save only those piece of data that needs to be serialized, and recreate the objects back into the memory upon loading. If it is done properly, then you don't have such thing as separate entity description used for serialization. The idea of such a system is primarily to reduce the programmer's time spend on thinking and implementing serialization effectively to 0. Just make the entities work inside the editor, and mark which members you want serialized. Alen |