Thread: Re: [Algorithms] Data base choices
Brought to you by:
vexxed72
From: Conor S. <bor...@ya...> - 2007-04-03 10:53:22
|
You beat me to it, I wrote a message just like this. I would make the point that serialization is really just writing out composable atoms in an order defined by a particular graph traversal of your object model. This means that you can do it to XML or you can do it to a database. If you have a serialization routine that supports optional names (or you're content with the nightmare that is matching up column numbers in a database), mark the beginning and ending of objects and abstract the method of writing out atoms, you can have a complete system that will allow you to take data out of a database and turn it into a tight piece of binary. Obviously if you start off with stuff in a database, then move to binary for your export... well then the sky is the limit. Also, serialization can be designed to work without complete data - so you can easily use asynchronous IO and have multiple buffers in flight while you read data. If you use a controlled data management process, you can create a consistent set of data and code written out from your database at a particular time and written to a nice set of fast to read binary files for each platform you target. I've worked on systems that do that multiple times a day servicing thousands of devices and over a dozen different platforms (many of which were tiny devices). Cheers, Conor ----- Original Message ---- From: Joris Mans <jor...@10...> To: Game Development Algorithms <gda...@li...> Sent: Tuesday, April 3, 2007 3:32:41 PM Subject: Re: [Algorithms] Data base choices I don't completely agree with your serialization issues here :) > 1) serialization is often not very portable Depends on how you implement it. The system we have overhere is works flawlessly on several platforms, including those with different endian-ness. We can save on any platform and load on any other > 2) serialization typically is binary, and thus version controls very > poorly This is correct, but we use binary serialization for the platform-specific data (e.g. levels). The cross platform editor data is stored in a text format, and the entities are created by script. And if you talk about loading old versions of files (if classes have changed in the code), that also is handled by our system (within certain limits ofcourse). > 3) you can't run reports on serialization What do you mean by that? > 4) it's hard to get just small sub-chunks out of a serialized stream Again, it depends on how you organize your serialization system > 5) it's really hard to open the data without the code -- and when you're > in time crunch, this can bite you I agree, but as said above, its not used for editor assets, only for what comes out the resource compiler > 6) serialization seldom lends itself to proper storage in RAM (i e, > you'll get the two-copy syndrome) > Again, if you do it without too much thinking I agree, but (sounding like a broken record here) overhere it works just fine and we can load data in-place. It works without issues when loading on consoles from DVD or CD. The disadvantage of not using a serialization system is that you have to write save/load code for every class/struct you want to store. All imho ofcourse :) Joris ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list ____________________________________________________________________________________ Looking for earth-friendly autos? Browse Top Cars by "Green Rating" at Yahoo! Autos' Green Center. http://autos.yahoo.com/green_center/ |
From: Nick T. <nic...@co...> - 2007-04-03 15:08:12
|
=0D=0ASelf serialising code that is evolving, by definition, will lead to=0D= =0Aproblems with backwards compatibility. You'll end up doing (more) work=0D= =0Ato patch up and use intermediate stored data (and it's a can of worms=0D= =0Awhen it come to writing tools to edit this data). I agree with what Jon=0D= =0Ais saying. A separate data schema is better for development. You can=0D=0A= always bake your intermediate/development format into something more=0D=0Ae= fficient for the final game. It need not be more complicated than CSV.=0D=0A= =20=0D=0AMoving away from self serialisation/initialisation to "object crea= tors"=0D=0Amay help. By this I mean given a data description [1] a factory = class=0D=0Awill create an object, stick it in the world and give it back to= you.=0D=0A=0D=0AI'd suggest switching from Boost serialisation to Boost Py= thon. Bind=0D=0Ayour object creator data descriptions (OCDD). In C++ you mi= ght supply a=0D=0Asimple map wrapper class (with key-value setter methods).= In Python you=0D=0Amight just take a dictionary and populate the OCDD (thr= ough your C++=0D=0Awrapper=3F) (which is easy using Boost Python).=0D=0A=0D= =0ANow to create an object you have to populate either a C++ OCDD or a=0D=0A= Python dictionary and pass this to the object manager which passes it to=0D= =0Athe correct factory. Once you have your object back you can wire up and=0D= =0Aevents (if this isn't in the OCDD).=0D=0A=0D=0ATo supply undo you just r= emember what actions have been applied to which=0D=0AOCDDs (and enough data= to reverse the change). You might just store an=0D=0Aediting session as a = list of delta commands (e.g. "set object 37 key X=0D=0Ato Y") which you can= apply after then editing session is finished [2].=0D=0A=0D=0AHow and where= you store the OCDD info is up to you. Having an XML=0D=0Adescription popul= ate a dictionary isn't a bad start. For release you=0D=0Amight bake all the= XML info into a binary block and hash all the keys=0D=0A[3] etc.=0D=0A=0D=0A= BTW, I'm sure if you go back over the history of this list and the sweng=0D= =0Alist you'll find plenty more reference on serialisation.=0D=0A=0D=0ANick=0D= =0A=0D=0A[1] This could very simply be name-value key-value pairs, e.g. std= ::map<=0D=0Astd::string, VariantType >. See Boost for variant type info. Th= e object=0D=0Acreators take these values and initialise your objects etc. B= ecause you=0D=0Aare using a dictionary and default values in your object yo= u have=0D=0Aforward and backward compatibility (as long as you are careful = with=0D=0Anames and value types)=0D=0A=0D=0A[2] There is nothing to stop yo= u writing this information across a=0D=0Anetwork and maintaining current st= ate in a remote tool.=0D=0A=0D=0A[3] These are all just suggestions, the im= plementation details are=0D=0Aobviously up to you.=0D=0A=0D=0A=0D=0A-----Or= iginal Message-----=0D=0AFrom: gda...@li...urceforge.= net=0D=0A[mailto:gda...@li...] On Behalf= Of=0D=0ADavid Neubelt=0D=0ASent: Tuesday, April 03, 2007 2:57 AM=0D=0ATo: = Game Development Algorithms=0D=0ASubject: [Algorithms] Data base choices=0D= =0A=0D=0AHey everyone,=0D=0A=0D=0A=09Currently, we are using boost serializ= ation and a fixed=0D=0Adirectory structure to represent our data base. Boos= t serialization=0D=0Ainvolves wrapping macros around each member variable a= nd writing a=0D=0Aserialize function. The benefit of boost is that it has v= ersion control,=0D=0Ameaning if you change the format of the class (say add= another member=0D=0Avariable) then all older version files on disc will st= ill load properly.=0D=0AIt also allows us a very quick way to save out an i= ntermediate version=0D=0Aof a class for incremental builds of an asset.=0D=0A= =09=0D=0A=09There are many disadvantages though. The compile time boost=0D=0A= introduces is very high as well as the dependency of having to include=0D=0A= boost serialization code in every class we want to serialize. Not only=0D=0A= that but we have boost serialization code that gets trickled through=0D=0Ae= very data structure we use. Another disadvantage is that for our=0D=0Ascrip= ting language to interface with it, currently, we have to write a=0D=0Ac++ = plug-in that takes a string request of the member variable name and=0D=0Are= turns its value. This has become a very tedious task and a bottleneck=0D=0A= when a scripter wants to get more access to the database. The last=0D=0Adis= advantage is that boost serialization process is very slow. Our build=0D=0A= times can grow very high if our database entries are large.=0D=0A=0D=0A=09W= ith the next version of our engine and tools we'd like to use a=0D=0Adataba= se that both python and c++ can interface into. It also must=0D=0Asupport v= ersion changes to the format and be fast (or at least no slower=0D=0Athen b= oost). Also, the database must allow for local changes that are=0D=0Arevert= able. Currently, everyone maintains a copy of the database on=0D=0Atheir ma= chine that they can muck around with and if they are satisfied=0D=0Athey ca= n commit there changes.=0D=0A=0D=0A=09I'm new to database development and p= rogramming so any advice is=0D=0Aalso appreciated.=0D=0A=0D=0A-=3D Dave=0D=0A=0D= =0A------------------------------------------------------------------------=0D= =0A-=0D=0ATake Surveys. Earn Cash. Influence the Future of IT=0D=0AJoin Sou= rceForge.net's Techsay panel and you'll get the chance to share=0D=0Ayour=0D= =0Aopinions on IT & business topics through brief surveys-and earn cash=0D=0A= http://www.techsay.com/default.php=3Fpage=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDE=0D=0AV=0D=0A=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=0D=0AGDAlgorithms-list mailing list=0D=0AGDAlgorithms-list@lists.= sourceforge.net=0D=0Ahttps://lists.sourceforge.net/lists/listinfo/gdalgorit= hms-list=0D=0AArchives:=0D=0Ahttp://sourceforge.net/mailarchive/forum.php=3F= forum=5Fname=3Dgdalgorithms-lis=0D=0At=0D=0A*******************************= ***************************************************=0D=0ADisclaimer=0D=0A=0D= =0AThe information and attached documentation in this e-mail is intended fo= r the use of the addressee only and is confidential. If you are not the int= ended recipient please delete it and notify us immediately by telephoning o= r e-mailing the sender. Please note that without Codemasters=E2=80=99 prior= written consent any form of distribution, copying or use of this communica= tion or the information in it is strictly prohibited and may be unlawful. =0D= =0A=0D=0AAttachments to this e-mail may contain software viruses. You are a= dvised to take all reasonable precautions to minimise this risk and to carr= y out a virus check on any documents before they are opened.=0D=0A=0D=0AAny= offer contained in this communication is subject to Codemasters=E2=80=99 s= tandard terms & conditions and must be signed by both parties. Except as ex= pressly provided otherwise all information and attached documentation in th= is e-mail is subject to contract and Codemasters=E2=80=99 board approval.=0D= =0AAny views or opinions expressed are solely those of the author and do no= t necessarily represent those of Codemasters.=0D=0A=0D=0AThis footnote also= confirms that this email message has been swept by=0D=0ASurfControl for th= e presence of computer viruses.=0D=0A**************************************= ********************************************=0D=0A |
From: Jamie F. <ja...@qu...> - 2007-04-03 15:26:50
|
Nick Trout wrote: > Self serialising code that is evolving, by definition, will lead to > problems with backwards compatibility. You'll end up doing (more) work > to patch up and use intermediate stored data (and it's a can of worms > when it come to writing tools to edit this data). I agree with what Jon > is saying. A separate data schema is better for development. You can > always bake your intermediate/development format into something more > efficient for the final game. It need not be more complicated than CSV. Your point is only really true if you interpret serializing an object as naively storing every member field, which I wouldn't. As far as I can see, your approach simply moves the problem, it doesn't fix it: that intermediate / development format is still a serialization of the object in some form (it's enough to reconstruct it, after all), and when you change that data format, you still have the same problem as you see in serializing a changing object. > Moving away from self serialisation/initialisation to "object creators" > may help. By this I mean given a data description [1] a factory class > will create an object, stick it in the world and give it back to you. > > I'd suggest switching from Boost serialisation to Boost Python. Bind > your object creator data descriptions (OCDD). In C++ you might supply a > simple map wrapper class (with key-value setter methods). In Python you > might just take a dictionary and populate the OCDD (through your C++ > wrapper?) (which is easy using Boost Python). > > Now to create an object you have to populate either a C++ OCDD or a > Python dictionary and pass this to the object manager which passes it to > the correct factory. Once you have your object back you can wire up and > events (if this isn't in the OCDD). > > To supply undo you just remember what actions have been applied to which > OCDDs (and enough data to reverse the change). You might just store an > editing session as a list of delta commands (e.g. "set object 37 key X > to Y") which you can apply after then editing session is finished [2]. > > How and where you store the OCDD info is up to you. Having an XML > description populate a dictionary isn't a bad start. For release you > might bake all the XML info into a binary block and hash all the keys > [3] etc. I can't see how this is any different to an object which serializes / deserializes itself: you've simply moved the responsibility for that action to another object. Jamie > BTW, I'm sure if you go back over the history of this list and the sweng > list you'll find plenty more reference on serialisation. > Nick > > [1] This could very simply be name-value key-value pairs, e.g. std::map< > std::string, VariantType >. See Boost for variant type info. The object > creators take these values and initialise your objects etc. Because you > are using a dictionary and default values in your object you have > forward and backward compatibility (as long as you are careful <snip> Here is one reappearance of the original versioning problem. </snip> with > names and value types) > > [2] There is nothing to stop you writing this information across a > network and maintaining current state in a remote tool. > > [3] These are all just suggestions, the implementation details are > obviously up to you. > > > -----Original Message----- > From: gda...@li... > [mailto:gda...@li...] On Behalf Of > David Neubelt > Sent: Tuesday, April 03, 2007 2:57 AM > To: Game Development Algorithms > Subject: [Algorithms] Data base choices > > Hey everyone, > > Currently, we are using boost serialization and a fixed > directory structure to represent our data base. Boost serialization > involves wrapping macros around each member variable and writing a > serialize function. The benefit of boost is that it has version control, > meaning if you change the format of the class (say add another member > variable) then all older version files on disc will still load properly. > It also allows us a very quick way to save out an intermediate version > of a class for incremental builds of an asset. > > There are many disadvantages though. The compile time boost > introduces is very high as well as the dependency of having to include > boost serialization code in every class we want to serialize. Not only > that but we have boost serialization code that gets trickled through > every data structure we use. Another disadvantage is that for our > scripting language to interface with it, currently, we have to write a > c++ plug-in that takes a string request of the member variable name and > returns its value. This has become a very tedious task and a bottleneck > when a scripter wants to get more access to the database. The last > disadvantage is that boost serialization process is very slow. Our build > times can grow very high if our database entries are large. > > With the next version of our engine and tools we'd like to use a > database that both python and c++ can interface into. It also must > support version changes to the format and be fast (or at least no slower > then boost). Also, the database must allow for local changes that are > revertable. Currently, everyone maintains a copy of the database on > their machine that they can muck around with and if they are satisfied > they can commit there changes. > > I'm new to database development and programming so any advice is > also appreciated. > > -= Dave |
From: Jon W. <hp...@mi...> - 2007-04-03 16:21:16
|
Jamie Fowlston wrote: > As far as I can see, your approach simply moves the problem, it doesn't > fix it: that intermediate / development format is still a serialization > of the object in some form (it's enough to reconstruct it, after all), > and when you change that data format, you still have the same problem as > you see in serializing a changing object. > I think that, in the end, we all end up with rich meta-data, that allows you to at a minimum introspect, and perhaps also reflect (useful for editors), and we build our I/O on top of the meta-data that is available. Whether that I/O is binary blobs or structured to a schema is then "less important" from an object point of view (but, I would argue, still important from a process point of view). So, when I say I dislike "serialization," I mean the traditional exhale/inhale style binary serialization, popularized by various UI toolkits and serialized object databases. If you solve all the problems, and end up with something very unlike that kind of serialization, you're in a much better place. But take a random guy from the industry, tell him to describe "serialization," and see what comes out! Cheers, / h+ |
From: Mat N. (BUNGIE) <Mat...@mi...> - 2007-04-03 16:43:41
|
Reflection is very important for both editors and flexible data analysis to= ols. If you also roll in data dependencies as a first-class citizen of your= data system, you can pretty much do whatever you want with your data. MSN -----Original Message----- From: gda...@li... [mailto:gdalgorithms-= lis...@li...] On Behalf Of Jon Watte Sent: Tuesday, April 03, 2007 9:21 AM To: Game Development Algorithms Subject: Re: [Algorithms] Data base choices Jamie Fowlston wrote: > As far as I can see, your approach simply moves the problem, it doesn't > fix it: that intermediate / development format is still a serialization > of the object in some form (it's enough to reconstruct it, after all), > and when you change that data format, you still have the same problem as > you see in serializing a changing object. > I think that, in the end, we all end up with rich meta-data, that allows you to at a minimum introspect, and perhaps also reflect (useful for editors), and we build our I/O on top of the meta-data that is available. Whether that I/O is binary blobs or structured to a schema is then "less important" from an object point of view (but, I would argue, still important from a process point of view). So, when I say I dislike "serialization," I mean the traditional exhale/inhale style binary serialization, popularized by various UI toolkits and serialized object databases. If you solve all the problems, and end up with something very unlike that kind of serialization, you're in a much better place. But take a random guy from the industry, tell him to describe "serialization," and see what comes out! Cheers, / h+ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share you= r opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE= VDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=3Dgdalgorithms-list |
From: Jamie F. <ja...@qu...> - 2007-04-03 17:22:52
|
Jon Watte wrote: > > Jamie Fowlston wrote: >> As far as I can see, your approach simply moves the problem, it doesn't >> fix it: that intermediate / development format is still a serialization >> of the object in some form (it's enough to reconstruct it, after all), >> and when you change that data format, you still have the same problem as >> you see in serializing a changing object. >> > > I think that, in the end, we all end up with rich meta-data, that allows > you to at a minimum introspect, and perhaps also reflect (useful for > editors), and we build our I/O on top of the meta-data that is > available. Rich metadata (in various flavours) certainly seems to be at the end of many roads. As all game objects built with our tech work seamlessly within the editor, we rely more upon runtime reflected objects than data reflection for editing. > Whether that I/O is binary blobs or structured to a schema is > then "less important" from an object point of view (but, I would argue, > still important from a process point of view). No mechanism can stop somebody storing an object badly if they want to. Obviously you should only store the real data needed to reconstruct the object. > So, when I say I dislike "serialization," I mean the traditional > exhale/inhale style binary serialization, popularized by various UI > toolkits and serialized object databases. If you solve all the problems, > and end up with something very unlike that kind of serialization, you're > in a much better place. But take a random guy from the industry, tell > him to describe "serialization," and see what comes out! Yes, naive serialization is bad. Naive _anything_ is almost always bad :) Jamie |
From: Nick T. <nic...@co...> - 2007-04-03 16:11:10
|
> Jamie Fowlston=0D=0A> Your point is only really true if you interpret ser= ializing an object as=20=0D=0Anaively storing every member field, which I w= ouldn't.=0D=0A=0D=0AI'm not suggesting serialising the object. I'm suggesti= ng serialising the data that is used to create the object. If you serialise= an object in order to create it, quite often what happens is you end up wi= th data that was only relevant during creation. Or you want certain creatio= n data that to translate into game values at creation time. Or the objects = are refactored but the creation data is essentially the same.=0D=0A=0D=0AI'= m suggesting creating an object which contains everything you need to know = for an object of that type to be created with all the correct initialisatio= n values. I'm suggesting that a factory creates the objects and perhaps use= s this serialisation object, or perhaps the class itself uses it. So you di= sconnect the creation data and the object data. (Like a fa=C3=A7ade pattern= for creating objects)=0D=0A=0D=0ABecause the creation data is stored in a = dictionary (and have default values) it solves the missing or extra data va= lues from incompatible versions. (i.e. its not just a blind binary block)=0D= =0A=0D=0A> JF:=0D=0A> As far as I can see, your approach simply moves the p= roblem, it doesn't=20=0D=0Afix it: that intermediate / development format i= s still a serialization=20=0D=0Aof the object in some form (it's enough to = reconstruct it, after all),=20=0D=0Aand when you change that data format, y= ou still have the same problem as=20=0D=0Ayou see in serializing a changing= object.=0D=0A=0D=0AWell, you completely pull an object apart and rewrite i= t in a different form its very likely that it's creation data will change! = If you are just refactoring systems though and the object creation data set= is the same then you don't need to refactor its creation data.=0D=0A=0D=0A= What are you suggesting then=3F Some sort of category list/meta-data in an = objects data so if you refactor the code you can refactor the data offline=3F=0D= =0A=0D=0A=0D=0ANT > How and where you store the OCDD info is up to you. Hav= ing an XML=0D=0A> description populate a dictionary isn't a bad start. For = release you=0D=0A> might bake all the XML info into a binary block and hash= all the keys=0D=0A> [3] etc.=0D=0A=0D=0AJF > I can't see how this is any d= ifferent to an object which serializes /=20=0D=0Adeserializes itself: you'v= e simply moved the responsibility for that=20=0D=0Aaction to another object= =2E=0D=0A=0D=0AYes that's what I've done. I don't know if you're reading fa= r more complexity into the original post=3F Or maybe I missed something=3F = :) I'm reading the post problems to be solved as:=0D=0A=0D=0A * Boost seria= lisation slow.=0D=0A * Boost serialisation not a good way of accessing data= values in game/exposing to script.=0D=0A * C++ and Python access required.=0D= =0A * Local change management required.=0D=0A * Object data members change = over time.=0D=0A=0D=0AYou seem to be implying there is massive refactoring = involved (=3F) whereas I'm not just suggesting another engineering solution= to the problem. I acknowledge the point you are making though, if you heav= ily refactor objects the creation data will likely change. However, to vent= ure into refactoring game data through metadata would seem to be one of tho= se "back to the drawing board this is way too complex" moments. :)=0D=0A=0D= =0ANick=0D=0A=0D=0A=0D=0A**************************************************= ********************************=0D=0ADisclaimer=0D=0A=0D=0AThe information= and attached documentation in this e-mail is intended for the use of the a= ddressee only and is confidential. If you are not the intended recipient pl= ease delete it and notify us immediately by telephoning or e-mailing the se= nder. Please note that without Codemasters=E2=80=99 prior written consent a= ny form of distribution, copying or use of this communication or the inform= ation in it is strictly prohibited and may be unlawful.=20=0D=0A=0D=0AAttac= hments to this e-mail may contain software viruses. You are advised to take= all reasonable precautions to minimise this risk and to carry out a virus = check on any documents before they are opened.=0D=0A=0D=0AAny offer contain= ed in this communication is subject to Codemasters=E2=80=99 standard terms = & conditions and must be signed by both parties. Except as expressly provid= ed otherwise all information and attached documentation in this e-mail is s= ubject to contract and Codemasters=E2=80=99 board approval.=0D=0AAny views = or opinions expressed are solely those of the author and do not necessarily= represent those of Codemasters.=0D=0A=0D=0AThis footnote also confirms tha= t this email message has been swept by=0D=0ASurfControl for the presence of= computer viruses.=0D=0A***************************************************= *******************************=0D=0A |
From: Jamie F. <ja...@qu...> - 2007-04-03 17:08:15
|
Nick Trout wrote: >> Jamie Fowlston >> Your point is only really true if you interpret serializing an object as > naively storing every member field, which I wouldn't. > > I'm not suggesting serialising the object. I'm suggesting serialising the data that is used to create the object. OK, that's what I'm suggesting too, and I don't see a difference between them :) > If you serialise an object in order to create it, quite often what happens is you end up with data that was only relevant during creation. Or you want certain creation data that to translate into game values at creation time. Or the objects are refactored but the creation data is essentially the same. > > I'm suggesting creating an object which contains everything you need to know for an object of that type to be created with all the correct initialisation values. I'm suggesting that a factory creates the objects and perhaps uses this serialisation object, or perhaps the class itself uses it. So you disconnect the creation data and the object data. (Like a façade pattern for creating objects) Intrinsically I just don't see the difference between what you're describing and how I think of (and use) serialization. > Because the creation data is stored in a dictionary (and have default values) it solves the missing or extra data values from incompatible versions. (i.e. its not just a blind binary block) This can be done just as easily by the object when using serialization. And your binary blocks don't have to be blind. I just see the same operations being done either way. You can move around where you store the data and which objects have responsibility for restoring the object, but it doesn't change what you're fundamentally doing. >> JF: >> As far as I can see, your approach simply moves the problem, it doesn't > fix it: that intermediate / development format is still a serialization > of the object in some form (it's enough to reconstruct it, after all), > and when you change that data format, you still have the same problem as > you see in serializing a changing object. > > Well, you completely pull an object apart and rewrite it in a different form its very likely that it's creation data will change! If you are just refactoring systems though and the object creation data set is the same then you don't need to refactor its creation data. Why? I still claim these are the same operation, whether it's "serialization" or based on a "data schema". You have objects you want to save and restore. You need to save some data. The data you need to save depends on what data you can assume on reconstruction. On reload, you have the default data and you patch in the new values. There's nothing intrinsically linked to either method in this abstract description of the problem, as far as I can see. > What are you suggesting then? Some sort of category list/meta-data in an objects data so if you refactor the code you can refactor the data offline? That wasn't really what I was suggesting, although I suppose you could do that with our system if you really wanted. In our system, each object serializes the various data (each with a name, which the backend may or may not store) it needs to restore itself. That stream has a version number. On reload, some fields of the object may already have default values from the constructor (or another default initializer... or it could be keyed off a "default values" field in the stream), then the object interprets its chunk of the stream to finish restoring itself. > NT > How and where you store the OCDD info is up to you. Having an XML >> description populate a dictionary isn't a bad start. For release you >> might bake all the XML info into a binary block and hash all the keys >> [3] etc. > > JF > I can't see how this is any different to an object which serializes / > deserializes itself: you've simply moved the responsibility for that > action to another object. > > Yes that's what I've done. I don't know if you're reading far more complexity into the original post? Or maybe I missed something? :) I'm reading the post problems to be solved as: > > * Boost serialisation slow. > * Boost serialisation not a good way of accessing data values in game/exposing to script. > * C++ and Python access required. > * Local change management required. > * Object data members change over time. > > You seem to be implying there is massive refactoring involved (?) whereas I'm not just suggesting another engineering solution to the problem. I acknowledge the point you are making though, if you heavily refactor objects the creation data will likely change. However, to venture into refactoring game data through metadata would seem to be one of those "back to the drawing board this is way too complex" moments. :) I may not be relating very well to the OP's point (I certainly don't know anything about Boost serialization :) I'm simply objecting to "serialization = bad", which seemed to be the main point of your post (when condensed to two words!) In the end, I think we probably agree...? Jamie > Nick |
From: Nick T. <nic...@co...> - 2007-04-04 10:03:45
|
> Jamie Fowlston:=0D=0A> Intrinsically I just don't see the difference betw= een what you're=20=0D=0Adescribing and how I think of (and use) serializati= on.=0D=0A=0D=0AMy answer is in the context of the original post. I'm trying= to answer=0D=0Athe question(s) I see posed there. I haven't used Boost's s= erialisation=0D=0Acode either but a quick glance at the overview and what i= s mentioned in=0D=0Athe post suggest that a different approach is required = (with script=0D=0Abinding in mind).=0D=0A=0D=0A> JF:=0D=0A> I may not be re= lating very well to the OP's point (I certainly don't=20=0D=0Aknow anything= about Boost serialization :) I'm simply objecting to=20=0D=0A"serializatio= n =3D bad", which seemed to be the main point of your post=20=0D=0A(when co= ndensed to two words!)=0D=0A=0D=0AIf you're a big fan of "exhale/inhale sty= le binary serialization" then=0D=0Athat's fine. It depends on your workflow= and games save points whether=0D=0Athat is a good idea or not. If your gam= e had "highest level achieved" as=0D=0Athe save point then an entire serial= isation system would seem somewhat=0D=0Aredundant and a perhaps unnecessary= over complication.=0D=0A=0D=0A> In the end, I think we probably agree...=3F=0D= =0A=0D=0A"If two men agree on everything, you may be sure that one of them = is=0D=0Adoing the thinking." -- Lyndon B. Johnson (1908 - 1973)=0D=0A=0D=0A= "The most likely way for the world to be destroyed, most experts agree,=0D=0A= is by accident. That's where we come in; we're computer professionals.=0D=0A= We cause accidents." -- Nathaniel Borenstein (1957 - )=0D=0A=0D=0ANick=0D=0A=0D= =0AComputer Professional=0D=0A=0D=0A=0D=0A*********************************= *************************************************=0D=0ADisclaimer=0D=0A=0D=0A= The information and attached documentation in this e-mail is intended for t= he use of the addressee only and is confidential. If you are not the intend= ed recipient please delete it and notify us immediately by telephoning or e= -mailing the sender. Please note that without Codemasters=E2=80=99 prior wr= itten consent any form of distribution, copying or use of this communicatio= n or the information in it is strictly prohibited and may be unlawful.=20=0D= =0A=0D=0AAttachments to this e-mail may contain software viruses. You are a= dvised to take all reasonable precautions to minimise this risk and to carr= y out a virus check on any documents before they are opened.=0D=0A=0D=0AAny= offer contained in this communication is subject to Codemasters=E2=80=99 s= tandard terms & conditions and must be signed by both parties. Except as ex= pressly provided otherwise all information and attached documentation in th= is e-mail is subject to contract and Codemasters=E2=80=99 board approval.=0D= =0AAny views or opinions expressed are solely those of the author and do no= t necessarily represent those of Codemasters.=0D=0A=0D=0AThis footnote also= confirms that this email message has been swept by=0D=0ASurfControl for th= e presence of computer viruses.=0D=0A**************************************= ********************************************=0D=0A |
From: Jamie F. <ja...@qu...> - 2007-04-04 10:32:26
|
Nick Trout wrote: >> JF: >> I may not be relating very well to the OP's point (I certainly don't > know anything about Boost serialization :) I'm simply objecting to > "serialization = bad", which seemed to be the main point of your post > (when condensed to two words!) > > If you're a big fan of "exhale/inhale style binary serialization" then > that's fine. No, I'm not, and I hope it's clear that the system I've described is nothing like that :) I think we're all agreed there is a distinction between "exhale/inhale" (or naive) serialization and the more sane and structured systems we've described. Jamie |
From: Tom P. <ga...@fa...> - 2007-04-04 23:34:51
|
Jamie Fowlston wrote: > > I may not be relating very well to the OP's point (I certainly don't=20 > > know anything about Boost serialization :) I'm simply objecting to=20 > > "serialization =3D bad", which seemed to be the main point of your = post=20 > > (when condensed to two words!) > >=20 > > If you're a big fan of "exhale/inhale style binary serialization" = then > > that's fine.=20 >=20 > No, I'm not, and I hope it's clear that the system I've described is=20 > nothing like that :) I think we're all agreed there is a distinction=20 > between "exhale/inhale" (or naive) serialization and the more sane and=20 > structured systems we've described. I don't see the distinction, frankly. Although my thoughts on the topic initially were that "serialization" equated to "loading and storing stuff from persistent media" and didn't necessarily imply that reads from storage were done field-wise. Everything I've read here, it really sounds like everyone ultimately ends up with the same thing, and I don't see much in the way of actual distinction except for the actual persistence mechanism (structured database vs. flat files). I do wish to throw out that I've authored several systems that allow most objects to be forward and backward compatible, but those all relied on the data being text; i.e. the schema was baked into the data. I could see such a mechanism working with binary blobs, too, though. Looking back through the thread, though, is "inhale/exhale" meant to imply every member of every structure is serialized in order, whereas "not inhale/exhale" means only the data that is required to reconstitute the object is stored? I'm firmly in the old-school camp, where you do a single big read and your whole world is "up". Then you hit your save game data, one read into one memory location and wham it's restored. I understand that this is hostile to versioning, although in my experience on consoles this has never been a problem. I recognize though that the online presence of consoles now will be changing that, however. At the same time, I still feel the idea has merit; just stream a big data block off of disc and be running. The near-zero CPU load streaming of terrain, for instance, made for some really large levels with no "loading please wait" prompts on-screen. The more CPU you need to use after loading your data, the less likely you'll be able to implement a seamless streaming solution. This point may ultimately be irrelevant to the discussion taking place, however, if folks don't care about streaming. :) -tom! --=20 |
From: Joris M. <jor...@10...> - 2007-04-05 00:52:06
|
Might I just add that with DVD drives and the such, much more time is actually wasted transferring the data from your medium to console/pc memory, and that you have heaps of cpu (now even more with multi-core systems) to do any CPU processing you might need on your loaded data whilst doing async I/O. You have to really do your best to make your loading code so slow that it processes bytes slower than your DVD drive can give you imho :) > running. The near-zero CPU load streaming of terrain, for instance, > made for some really large levels with no "loading please wait" prompts > on-screen. The more CPU you need to use after loading your data, the > less likely you'll be able to implement a seamless streaming solution. > This point may ultimately be irrelevant to the discussion taking place, > however, if folks don't care about streaming. :) > |
From: Pal-Kristian E. <pal...@na...> - 2007-04-05 01:49:54
|
Hi, I'm not sure I understand. Are you arguing that consoles are so fast nowadays that we can afford to waste CPU bandwidth and cycles on this? That doesn't make sense to me. Why spend developer time on something that wastes cycles as well as complicates your system at the core? Why not keep it simple and fast! PKE. Joris Mans wrote: > Might I just add that with DVD drives and the such, much more time is > actually wasted transferring the data from your medium to console/pc memory, > and that you have heaps of cpu (now even more with multi-core systems) to do > any CPU processing you might need on your loaded data whilst doing async > I/O. You have to really do your best to make your loading code so slow that > it processes bytes slower than your DVD drive can give you imho :) > > >> running. The near-zero CPU load streaming of terrain, for instance, >> made for some really large levels with no "loading please wait" prompts >> on-screen. The more CPU you need to use after loading your data, the >> less likely you'll be able to implement a seamless streaming solution. >> This point may ultimately be irrelevant to the discussion taking place, >> however, if folks don't care about streaming. :) >> >> > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list > > -- Pål-Kristian Engstad (en...@na...), Lead Programmer, ICE team, Naughty Dog, Inc., 1601 Cloverfield Blvd, 6000 North, Santa Monica, CA 90404, USA. Ph.: (310) 633-9112. "Most of us would do well to remember that there is a reason Carmack is Carmack, and we are not Carmack.", Jonathan Blow, 2/1/2006, GD Algo Mailing List |
From: Jamie F. <ja...@qu...> - 2007-04-05 13:04:50
|
Tom Plunket wrote: > Jamie Fowlston wrote: > >>> I may not be relating very well to the OP's point (I certainly don't >>> know anything about Boost serialization :) I'm simply objecting to >>> "serialization = bad", which seemed to be the main point of your post >>> (when condensed to two words!) >>> >>> If you're a big fan of "exhale/inhale style binary serialization" then >>> that's fine. >> No, I'm not, and I hope it's clear that the system I've described is >> nothing like that :) I think we're all agreed there is a distinction >> between "exhale/inhale" (or naive) serialization and the more sane and >> structured systems we've described. > > I don't see the distinction, frankly. Although my thoughts on the topic > initially were that "serialization" equated to "loading and storing > stuff from persistent media" and didn't necessarily imply that reads > from storage were done field-wise. My understanding of serialization is much like yours, and I think some of the confusion has come from others understanding serialization as an inhale/exhale thing, which is pretty much just blatting the memory the object occupies to disk (fixing pointers up on the way). Whilst the latter is a type of serialization, it's not one I'd choose to use any more (having done so many years ago). > Everything I've read here, it really sounds like everyone ultimately > ends up with the same thing, and I don't see much in the way of actual > distinction except for the actual persistence mechanism (structured > database vs. flat files). Did anybody argue for a flat file? > I do wish to throw out that I've authored several systems that allow > most objects to be forward and backward compatible, but those all relied > on the data being text; i.e. the schema was baked into the data. I > could see such a mechanism working with binary blobs, too, though. It can indeed work for binary blobs. Our system uses either text or binary depending upon which backend you choose. > Looking back through the thread, though, is "inhale/exhale" meant to > imply every member of every structure is serialized in order, whereas > "not inhale/exhale" means only the data that is required to reconstitute > the object is stored? My understanding is that an inhale/exhale system has the data in the file being as close to the runtime memory layout as possible. "Not inhale/exhale" covers everything else; which would usually mean you'd only store the data that you know you'd need to reconstruct the object. > I'm firmly in the old-school camp, where you do a single big read and > your whole world is "up". Then you hit your save game data, one read > into one memory location and wham it's restored. I understand that this > is hostile to versioning, although in my experience on consoles this has > never been a problem. I recognize though that the online presence of > consoles now will be changing that, however. At the same time, I still > feel the idea has merit; just stream a big data block off of disc and be > running. The near-zero CPU load streaming of terrain, for instance, > made for some really large levels with no "loading please wait" prompts > on-screen. The main advantage of this system is engineering simplicity, I would suggest. > The more CPU you need to use after loading your data, the > less likely you'll be able to implement a seamless streaming solution. A key part of our seamless streaming solution is a background work queue; one of its purposes is to prepare recently loaded data for use. This can include decompressing JPEGs, PNGs, or anything else which a plugin (could be ours, could be a custom one) chooses to do when it's loaded. > This point may ultimately be irrelevant to the discussion taking place, > however, if folks don't care about streaming. :) We care deeply about streaming :) Jamie |
From: <chr...@pl...> - 2007-04-05 18:06:31
|
Jamie Fowlston wrote to Tom Plunket: > > I'm firmly in the old-school camp, where you do a single big read and > > your whole world is "up". [...] > > The more CPU you need to use after loading your data, the > > less likely you'll be able to implement a seamless streaming solution. > > A key part of our seamless streaming solution is a background work > queue; one of its purposes is to prepare recently loaded data for use. > This can include decompressing JPEGs, PNGs, or anything else which a > plugin (could be ours, could be a custom one) chooses to do when it's > loaded. On God of War I/II we have both. Everything for a level sits in one big file that we load into memory through a ring buffer. This file consists of "chunks" of data, prefixed by a header. Each frame, we use some fixed percentage of the frame to process (complete) chunks in the ring buffer. The processing is sent to different code routines depending on what type of chunk it is. To make sure chunks are always linear in memory for processing, chunks that are too large to fit in the ring buffer, or that straddle the ring buffer boundary, we copy to scratch memory before processing. Benefits of this approach: * It is fast to load (a single straight file) * It allows unique decompression methods for each chunk type * It is pretty simple to implement * We can load-balance by controlling how much processing is done each frame Drawback: * We have a ring buffer sitting around in memory that's hard to use for other purposes Christer Ericson, Director of Tools and Technology Sony Computer Entertainment, Santa Monica |
From: <phi...@pl...> - 2007-04-06 02:29:36
|
Christer wrote: > Drawback: > > * We have a ring buffer sitting around in memory that's hard > to use for other purposes It was worse than that, we actually had two ring buffers. One on the IOP, and one on the EE. I could swear there was another 4k we could have gotten out of VU0, if we'd really tried. Cheers, Phil |
From: Jon W. <hp...@mi...> - 2007-04-06 06:08:18
|
chr...@pl... wrote: > what type of chunk it is. To make sure chunks are always linear in > memory for processing, chunks that are too large to fit in the ring buffer, > or that straddle the ring buffer boundary, we copy to scratch memory before > processing. > Do you have access to the MMU on the PS/3 like you had on the PS/2? If so, you can create a virtual space mapping that maps pages A, B, then A again into contiguous virtual space. This lets you treat the ring buffer as a contiguous block, even when a piece of data straddles a boundary, as long as no single data piece is bigger than half the ring buffer. Cheers, / h+ |
From: Tom P. <ga...@fa...> - 2007-04-05 01:37:58
|
> Might I just add that with DVD drives and the such, much more time > is actually wasted transferring the data from your medium to > console/pc memory, and that you have heaps of cpu (now even more > with multi-core systems) to do any CPU processing you might need on > your loaded data whilst doing async I/O. You have to really do your > best to make your loading code so slow that it processes bytes > slower than your DVD drive can give you imho :) Certainly; it depends on where you've got time. On opposite ends of the spectrum, you're either not fully utilizing your CPU or you have CPU to spare. On another (not-orthogonal) axis, you either need to load as fast as possible or you don't. In the cases with which I'm familiar, we had a moderately "full" CPU, so it was preferable to load more data since that could be scheduled to complete before the data was necessary. If, on the other hand, you need the data Right Now, and you have CPU to spare, by all means do what it takes to "compress" that data footprint. However, also remember that it's better to do a continuous read of one big block of data than it is to read a lot of small bits that are strewn hither and yon. ...at least, off of disc-based media, because seeking is Really Slow. "My" suggestion requires locality of data, whereas "yours" doesn't require it even if most everyone ends up trying to make it happen to some degree. So I'm not arguing with the idea of loading "prototype" (thx Charles) data off of disc as opposed to fully constituted memory images, but as with everything else in game development, it really depends on what one's trying to accomplish and the constraints under which one is working. -tom! |
From: Tom P. <ga...@fa...> - 2007-04-05 01:40:49
|
> On opposite ends of the spectrum, you're either not fully utilizing > your CPU or you have CPU to spare. I intended to fix that before posting, sadly I fixed both sides of the OR. So yeah, either you're using all your CPU or you ain't. Guess it should be obvious, but I am a clown so what do you expect? -tom! |
From: Joris M. <jor...@10...> - 2007-04-05 02:19:01
|
One can never waste bandwidth or cycles :) And keeping it 'simple and fast' really depends on your codebase. I'm just saying that making sure you can do sequential buffered reads from your disk is much more important than worrying about wasting some cycles of cpu to process the data you just read. ----- Original Message ----- From: "Pal-Kristian Engstad" <pal...@na...> To: "Game Development Algorithms" <gda...@li...> Sent: Thursday, April 05, 2007 3:48 AM Subject: Re: [Algorithms] Data base choices Hi, I'm not sure I understand. Are you arguing that consoles are so fast nowadays that we can afford to waste CPU bandwidth and cycles on this? That doesn't make sense to me. Why spend developer time on something that wastes cycles as well as complicates your system at the core? Why not keep it simple and fast! PKE. Joris Mans wrote: > Might I just add that with DVD drives and the such, much more time is > actually wasted transferring the data from your medium to console/pc > memory, > and that you have heaps of cpu (now even more with multi-core systems) to > do > any CPU processing you might need on your loaded data whilst doing async > I/O. You have to really do your best to make your loading code so slow > that > it processes bytes slower than your DVD drive can give you imho :) > > >> running. The near-zero CPU load streaming of terrain, for instance, >> made for some really large levels with no "loading please wait" prompts >> on-screen. The more CPU you need to use after loading your data, the >> less likely you'll be able to implement a seamless streaming solution. >> This point may ultimately be irrelevant to the discussion taking place, >> however, if folks don't care about streaming. :) >> >> > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list > > -- Pål-Kristian Engstad (en...@na...), Lead Programmer, ICE team, Naughty Dog, Inc., 1601 Cloverfield Blvd, 6000 North, Santa Monica, CA 90404, USA. Ph.: (310) 633-9112. "Most of us would do well to remember that there is a reason Carmack is Carmack, and we are not Carmack.", Jonathan Blow, 2/1/2006, GD Algo Mailing List ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list |
From: Conor S. <bor...@ya...> - 2007-04-05 02:42:32
|
No, he's saying that loading is an IO limited process. It can take signific= antly more time to do the IO than it does to do the loading once the IO is = done, so it can pay to use CPU time to reduce IO load. Even with asynchrono= us IO, the IO can also take a significantly higher amount of CPU time (on t= op of the seek time etc) than deserialisation code. Especially if you do re= ads into a single big buffer at a time that actually requires multiple read= s from the media behind the scenes. =0A=0AUsing "deserialize as you go" wit= h multiple sequential IO operations in flight can be faster than a single b= ig read. It can also solve problems involved with finding large pools of se= quential memory to do your reading - Populating an object graph through des= erialization lends itself naturally to quick region based allocation which = means you can get low levels of fragmentation with relatively simple and fa= st allocators.=0A=0AAnd you don't have the complexity of pointer fixing cod= e :-). Of course, all of this is only true in the cases where you are IO li= mited, or you IO operations have a reasonable impact on CPU time (in other = words, your mileage may vary case by case).=0A=0ACheers,=0AConor=0A=0A-----= Original Message ----=0AFrom: Pal-Kristian Engstad <pal_engstad@naughtydog= .com>=0ATo: Game Development Algorithms <gda...@li...urceforg= e.net>=0ASent: Thursday, April 5, 2007 9:48:31 AM=0ASubject: Re: [Algorithm= s] Data base choices=0A=0AHi,=0A=0AI'm not sure I understand. Are you argui= ng that consoles are so fast =0Anowadays that we can afford to waste CPU ba= ndwidth and cycles on this? =0AThat doesn't make sense to me. Why spend dev= eloper time on something =0Athat wastes cycles as well as complicates your = system at the core? Why =0Anot keep it simple and fast!=0A=0APKE.=0A=0AJori= s Mans wrote:=0A> Might I just add that with DVD drives and the such, much = more time is =0A> actually wasted transferring the data from your medium to= console/pc memory, =0A> and that you have heaps of cpu (now even more with= multi-core systems) to do =0A> any CPU processing you might need on your l= oaded data whilst doing async =0A> I/O. You have to really do your best to = make your loading code so slow that =0A> it processes bytes slower than you= r DVD drive can give you imho :)=0A>=0A> =0A>> running. The near-zero CP= U load streaming of terrain, for instance,=0A>> made for some really large = levels with no "loading please wait" prompts=0A>> on-screen. The more CPU = you need to use after loading your data, the=0A>> less likely you'll be abl= e to implement a seamless streaming solution.=0A>> This point may ultimatel= y be irrelevant to the discussion taking place,=0A>> however, if folks don'= t care about streaming. :)=0A>>=0A>> =0A>=0A>=0A> --------------------= -----------------------------------------------------=0A> Take Surveys. Ear= n Cash. Influence the Future of IT=0A> Join SourceForge.net's Techsay panel= and you'll get the chance to share your=0A> opinions on IT & business topi= cs through brief surveys-and earn cash=0A> http://www.techsay.com/default.p= hp?page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV=0A> _______________________= ________________________=0A> GDAlgorithms-list mailing list=0A> GDAlgorithm= s-...@li...=0A> https://lists.sourceforge.net/lists/listin= fo/gdalgorithms-list=0A> Archives:=0A> http://sourceforge.net/mailarchive/f= orum.php?forum_name=3Dgdalgorithms-list=0A>=0A> =0A=0A-- =0AP=E5l-Kristia= n Engstad (en...@na...), Lead Programmer, ICE=0Ateam, Naughty Do= g, Inc., 1601 Cloverfield Blvd, 6000 North,=0ASanta Monica, CA 90404, USA. = Ph.: (310) 633-9112.=0A=0A"Most of us would do well to remember that there = is a reason Carmack=0Ais Carmack, and we are not Carmack.",=0A = Jonathan Blow, 2/1/2006, GD Algo Mailing List=0A=0A=0A=0A=0A-----= --------------------------------------------------------------------=0ATake= Surveys. Earn Cash. Influence the Future of IT=0AJoin SourceForge.net's Te= chsay panel and you'll get the chance to share your=0Aopinions on IT & busi= ness topics through brief surveys-and earn cash=0Ahttp://www.techsay.com/de= fault.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDEVDEV=0A__________________= _____________________________=0AGDAlgorithms-list mailing list=0AGDAlgorith= ms...@li...=0Ahttps://lists.sourceforge.net/lists/listinf= o/gdalgorithms-list=0AArchives:=0Ahttp://sourceforge.net/mailarchive/forum.= php?forum_name=3Dgdalgorithms-list=0A=0A=0A=0A=0A=0A =0A___________________= _________________________________________________________________=0ALooking= for earth-friendly autos? =0ABrowse Top Cars by "Green Rating" at Yahoo! A= utos' Green Center.=0Ahttp://autos.yahoo.com/green_center/ |
From: Stephen W. <st...@wa...> - 2007-04-05 03:09:50
|
Well.. what a thread. Here I thought loading was simple. :) Anyway, just curious if any of you have any of your serialization (or whatever you all currently don't agree to call it) code public? --Steve |
From: Mat N. (BUNGIE) <Mat...@mi...> - 2007-04-05 03:46:20
|
Bah, pointer fixing isn't that much of a time waster. Maybe a slight memory= waste, but it isn't *that* bad. MSN -----Original Message----- From: gda...@li... [mailto:gdalgorithms-= lis...@li...] On Behalf Of Conor Stokes Sent: Wednesday, April 04, 2007 7:42 PM To: Game Development Algorithms Subject: Re: [Algorithms] Data base choices No, he's saying that loading is an IO limited process. It can take signific= antly more time to do the IO than it does to do the loading once the IO is = done, so it can pay to use CPU time to reduce IO load. Even with asynchrono= us IO, the IO can also take a significantly higher amount of CPU time (on t= op of the seek time etc) than deserialisation code. Especially if you do re= ads into a single big buffer at a time that actually requires multiple read= s from the media behind the scenes. Using "deserialize as you go" with multiple sequential IO operations in fli= ght can be faster than a single big read. It can also solve problems involv= ed with finding large pools of sequential memory to do your reading - Popul= ating an object graph through deserialization lends itself naturally to qui= ck region based allocation which means you can get low levels of fragmentat= ion with relatively simple and fast allocators. And you don't have the complexity of pointer fixing code :-). Of course, al= l of this is only true in the cases where you are IO limited, or you IO ope= rations have a reasonable impact on CPU time (in other words, your mileage = may vary case by case). Cheers, Conor ----- Original Message ---- From: Pal-Kristian Engstad <pal...@na...> To: Game Development Algorithms <gda...@li...> Sent: Thursday, April 5, 2007 9:48:31 AM Subject: Re: [Algorithms] Data base choices Hi, I'm not sure I understand. Are you arguing that consoles are so fast nowadays that we can afford to waste CPU bandwidth and cycles on this? That doesn't make sense to me. Why spend developer time on something that wastes cycles as well as complicates your system at the core? Why not keep it simple and fast! PKE. Joris Mans wrote: > Might I just add that with DVD drives and the such, much more time is > actually wasted transferring the data from your medium to console/pc memo= ry, > and that you have heaps of cpu (now even more with multi-core systems) to= do > any CPU processing you might need on your loaded data whilst doing async > I/O. You have to really do your best to make your loading code so slow th= at > it processes bytes slower than your DVD drive can give you imho :) > > >> running. The near-zero CPU load streaming of terrain, for instance, >> made for some really large levels with no "loading please wait" prompts >> on-screen. The more CPU you need to use after loading your data, the >> less likely you'll be able to implement a seamless streaming solution. >> This point may ultimately be irrelevant to the discussion taking place, >> however, if folks don't care about streaming. :) >> >> > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share y= our > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV > _______________________________________________ > GDAlgorithms-list mailing list > GDA...@li... > https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list > Archives: > http://sourceforge.net/mailarchive/forum.php?forum_name=3Dgdalgorithms-li= st > > -- P=E5l-Kristian Engstad (en...@na...), Lead Programmer, ICE team, Naughty Dog, Inc., 1601 Cloverfield Blvd, 6000 North, Santa Monica, CA 90404, USA. Ph.: (310) 633-9112. "Most of us would do well to remember that there is a reason Carmack is Carmack, and we are not Carmack.", Jonathan Blow, 2/1/2006, GD Algo Mailing List ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share you= r opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE= VDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=3Dgdalgorithms-list ___________________________________________________________________________= _________ Looking for earth-friendly autos? Browse Top Cars by "Green Rating" at Yahoo! Autos' Green Center. http://autos.yahoo.com/green_center/ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share you= r opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE= VDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=3Dgdalgorithms-list |
From: Jason H. <jas...@di...> - 2007-04-05 04:58:52
|
Conor Stokes wrote: > Even with asynchronous IO, the IO can also take a significantly higher amount of CPU time (on top of the seek time etc) than deserialisation code. Especially if you do reads into a single big buffer at a time that actually requires multiple reads from the media behind the scenes. > The question is whether the initialization stage is necessary at all. Big-block loading with fixups avoids it entirely, but not without some cost to the developer (development time, not runtime). It's harder to embed C++ objects with vtables in such things, but not impossible. And again, these are simply fixups that need to be made, nothing near as complex as a memory allocation. > Using "deserialize as you go" with multiple sequential IO operations in flight can be faster than a single big read. Extraordinary claims require extraordinary evidence... > It can also solve problems involved with finding large pools of sequential memory to do your reading - Populating an object graph through deserialization lends itself naturally to quick region based allocation which means you can get low levels of fragmentation with relatively simple and fast allocators. > ...and if the same objects are already embedded inside a single block, this is somehow inferior? If you know the exact size of each future load, there should be no trouble setting up your memory map to accomodate it. Using allocators, this is actually substantially harder without a handle-based defragging system, because pools might need to resize if they are shared across levels or chunks. I've written it both ways, many times. :-) > And you don't have the complexity of pointer fixing code :-). Of course, all of this is only true in the cases where you are IO limited, or you IO operations have a reasonable impact on CPU time (in other words, your mileage may vary case by case). > The complicated stuff is in the tools, and it's not that complicated. The runtime pointer fixup logic is literally: for (int i=0; i<numAddresses; i++) { *(baseAddr + fixup[i]) += baseAddr; } Oops, I think I gave away some proprietary secrets! ;-) JH |
From: Mark_Danks@PlayStation.Sony.Com - 2007-04-05 05:27:28
|
In reading this thread, it seems to have turned into two topics to me... One is how to represent dependencies with asset build systems (which is extremely interesting to me)...I'll comment with another subject line. The other is how to deal with serialized data (classes, objects, foobar, whatever). I'll mention how I've handled it for my $0.02. Previously, I've tried to differentiate between dev and shipping assets. Considering that I spend 90% of my time in dev (at least before I joined SCEA), I optimize for that. Shipping is the end stage. Shipping is also when I care the most about compression, load speeds off disc, etc. In dev, I care most about productivity for the other people on my team (not only programmers, but also level designers and artists). I tend to make sure that there is a nightly build for all content, but then there are local overrides by the production people. For object data (such as "serialized class structures"), I have done a key/value pair for the member variables. Yes, at load time, I have to fully construct the objects (ie, it isn't a binary load and go), but it gives a nice benefit. Because it is a key/value pair, the game can load the nightly build, and then load the local changes on top of it to get any modifications that an artist/designer might have done. With default values in code, you can also release binaries to the team without breaking all of your data. Is it inefficient for a shipping game? Heck, yes. Once I'm into shipping mode, I have the resource compilers generate binary blobs which can be blasted into the class data structures. Less backwards compatibility and robustness, but that isn't what I'm focusing on at that time. In reading all of the posts, it seems like people have different priorities, which set up different ways of doing things. If you want to be in a shipping mode, then parsing XML or text strings is evil. If you are focusing on the dev mode, then taking a performance or space hit is okay because it can increase productivity. Mark Danks Senior Manager, Dev Support SCEA Jason Hughes <jas...@di...> Sent by: gda...@li... 04/04/2007 09:58 PM Please respond to Game Development Algorithms <gda...@li...> To Game Development Algorithms <gda...@li...> cc Subject Re: [Algorithms] Data base choices Conor Stokes wrote: > Even with asynchronous IO, the IO can also take a significantly higher amount of CPU time (on top of the seek time etc) than deserialisation code. Especially if you do reads into a single big buffer at a time that actually requires multiple reads from the media behind the scenes. > The question is whether the initialization stage is necessary at all. Big-block loading with fixups avoids it entirely, but not without some cost to the developer (development time, not runtime). It's harder to embed C++ objects with vtables in such things, but not impossible. And again, these are simply fixups that need to be made, nothing near as complex as a memory allocation. > Using "deserialize as you go" with multiple sequential IO operations in flight can be faster than a single big read. Extraordinary claims require extraordinary evidence... > It can also solve problems involved with finding large pools of sequential memory to do your reading - Populating an object graph through deserialization lends itself naturally to quick region based allocation which means you can get low levels of fragmentation with relatively simple and fast allocators. > ...and if the same objects are already embedded inside a single block, this is somehow inferior? If you know the exact size of each future load, there should be no trouble setting up your memory map to accomodate it. Using allocators, this is actually substantially harder without a handle-based defragging system, because pools might need to resize if they are shared across levels or chunks. I've written it both ways, many times. :-) > And you don't have the complexity of pointer fixing code :-). Of course, all of this is only true in the cases where you are IO limited, or you IO operations have a reasonable impact on CPU time (in other words, your mileage may vary case by case). > The complicated stuff is in the tools, and it's not that complicated. The runtime pointer fixup logic is literally: for (int i=0; i<numAddresses; i++) { *(baseAddr + fixup[i]) += baseAddr; } Oops, I think I gave away some proprietary secrets! ;-) JH ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ GDAlgorithms-list mailing list GDA...@li... https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list Archives: http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list |