gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Friday, April 17, 2009, 9:59:01 PM, Mat wrote:
> You could always flip that around and have a to-C++ compiler.

And get tangled in a dependency quagmire. Used that a long time ago
and I like this direction much better. :)

Alen

Re: [Algorithms] Complexity of new hardware

[Mat N. <mat...@bu...>]

Do tell! Are you talking about cross dependencies between magic file<->C++?

MSN
-----Original Message-----
From: Alen Ladavac [mailto:ale...@cr...]
Sent: Friday, April 17, 2009 1:37 PM
To: Mat Noguchi
Cc: Game Development Algorithms; Jarkko Lempiainen
Subject: Re[2]: [Algorithms] Complexity of new hardware

Friday, April 17, 2009, 9:59:01 PM, Mat wrote:
> You could always flip that around and have a to-C++ compiler.

And get tangled in a dependency quagmire. Used that a long time ago
and I like this direction much better. :)

Alen

Re: [Algorithms] Complexity of new hardware

[Nicholas \Indy\ R. <ar...@gm...>]

I find that when parsing c++ it's to easy to write code that is
outside the scope of what you're parser can handle and it's much nicer
to do a to-C++ style compiler. The trick with that though is to ensure
that you're compiler does not depend on any code that it generates, if
it does, then you are in all sorts of hell.

Indy

On Fri, Apr 17, 2009 at 1:42 PM, Mat Noguchi <mat...@bu...> wrote:
> Do tell! Are you talking about cross dependencies between magic file<->C++?
>
> MSN
> -----Original Message-----
> From: Alen Ladavac [mailto:ale...@cr...]
> Sent: Friday, April 17, 2009 1:37 PM
> To: Mat Noguchi
> Cc: Game Development Algorithms; Jarkko Lempiainen
> Subject: Re[2]: [Algorithms] Complexity of new hardware
>
> Friday, April 17, 2009, 9:59:01 PM, Mat wrote:
>> You could always flip that around and have a to-C++ compiler.
>
> And get tangled in a dependency quagmire. Used that a long time ago
> and I like this direction much better. :)
>
> Alen
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Perhaps it depends on what you consider a to-C++ compiler. Cross-scope
type resolving is simpler if it is handled at startup time instead of
by the compiler.

Alen

Friday, April 17, 2009, 11:06:49 PM, Nicholas wrote:

> I find that when parsing c++ it's to easy to write code that is
> outside the scope of what you're parser can handle and it's much nicer
> to do a to-C++ style compiler. The trick with that though is to ensure
> that you're compiler does not depend on any code that it generates, if
> it does, then you are in all sorts of hell.

> Indy

> On Fri, Apr 17, 2009 at 1:42 PM, Mat Noguchi <mat...@bu...> wrote:
>> Do tell! Are you talking about cross dependencies between magic file<->C++?
>>
>> MSN
>> -----Original Message-----
>> From: Alen Ladavac [mailto:ale...@cr...]
>> Sent: Friday, April 17, 2009 1:37 PM
>> To: Mat Noguchi
>> Cc: Game Development Algorithms; Jarkko Lempiainen
>> Subject: Re[2]: [Algorithms] Complexity of new hardware
>>
>> Friday, April 17, 2009, 9:59:01 PM, Mat wrote:
>>> You could always flip that around and have a to-C++ compiler.
>>
>> And get tangled in a dependency quagmire. Used that a long time ago
>> and I like this direction much better.  
>>
>> Alen
>>
>>
>> ------------------------------------------------------------------------------
>> Stay on top of everything new and different, both inside and
>> around Java (TM) technology - register by April 22, and save
>> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
>> 300 plus technical and hands-on sessions. Register today.
>> Use priority code J9JMT32. http://p.sf.net/sfu/p
>> _______________________________________________
>> GDAlgorithms-list mailing list
>> GDA...@li...
>> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
>> Archives:
>> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>>

> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list



-- 
Best regards,
 Alen                            mailto:ale...@cr...

Re: [Algorithms] Complexity of new hardware

[Jarkko L. <al...@gm...>]

So far defining introspection straight in C++ has worked well for me, so you
don't have to generate nor parse anything and have full arsenal of C++
features in your use.


Cheers, Jarkko


-----Original Message-----
From: Nicholas "Indy" Ray [mailto:ar...@gm...] 
Sent: Saturday, April 18, 2009 12:07 AM
To: Game Development Algorithms
Subject: Re: [Algorithms] Complexity of new hardware

I find that when parsing c++ it's to easy to write code that is
outside the scope of what you're parser can handle and it's much nicer
to do a to-C++ style compiler. The trick with that though is to ensure
that you're compiler does not depend on any code that it generates, if
it does, then you are in all sorts of hell.

Indy

On Fri, Apr 17, 2009 at 1:42 PM, Mat Noguchi <mat...@bu...> wrote:
> Do tell! Are you talking about cross dependencies between magic
file<->C++?
>
> MSN
> -----Original Message-----
> From: Alen Ladavac [mailto:ale...@cr...]
> Sent: Friday, April 17, 2009 1:37 PM
> To: Mat Noguchi
> Cc: Game Development Algorithms; Jarkko Lempiainen
> Subject: Re[2]: [Algorithms] Complexity of new hardware
>
> Friday, April 17, 2009, 9:59:01 PM, Mat wrote:
>> You could always flip that around and have a to-C++ compiler.
>
> And get tangled in a dependency quagmire. Used that a long time ago
> and I like this direction much better. :)
>
> Alen
>
>
>
----------------------------------------------------------------------------
--
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>

----------------------------------------------------------------------------
--
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
GDAlgorithms-list mailing list
GDA...@li...
https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Jon W. <jw...@gm...>]

Jarkko Lempiainen wrote:
> So far defining introspection straight in C++ has worked well for me, so you
> don't have to generate nor parse anything and have full arsenal of C++
> features in your use.
>   

Yes; the only thing that is annoying is the repeating-yourself problem.
I find I usually need to repeat myself one and a half time.
First, it's the declaration.
Then, it's the repeat of the declaration, in macro form.
Finally, it's some small bits of macro that builds the 
helpers/visitors/registration needed to make it all "go."

Sincerely,

jw

Re: [Algorithms] Complexity of new hardware

[Jarkko L. <al...@gm...>]

Yeah, there is some repeating of the variable names, but it's not that bad
really. All I have is something like:

struct foo
{ PFC_MONO(foo) {PFC_VAR3(x, y, z);}
  int x, y, z;
};

And after that you can e.g. serialize or expose the class in editor:

foo f={1, 2, 3};
*open_file_write("hello.data")<<f;

I have used it for some time now and find it quite convenient.


Cheers, Jarkko


-----Original Message-----
From: Jon Watte [mailto:jw...@gm...] 
Sent: Saturday, April 18, 2009 5:27 PM
To: Game Development Algorithms
Subject: Re: [Algorithms] Complexity of new hardware

Jarkko Lempiainen wrote:
> So far defining introspection straight in C++ has worked well for me, so
you
> don't have to generate nor parse anything and have full arsenal of C++
> features in your use.
>   

Yes; the only thing that is annoying is the repeating-yourself problem.
I find I usually need to repeat myself one and a half time.
First, it's the declaration.
Then, it's the repeat of the declaration, in macro form.
Finally, it's some small bits of macro that builds the 
helpers/visitors/registration needed to make it all "go."

Sincerely,

jw


----------------------------------------------------------------------------
--
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
GDAlgorithms-list mailing list
GDA...@li...
https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Will V. <wi...@se...>]

On Sun, 19 Apr 2009 03:03:27 +1200, Jarkko Lempiainen <al...@gm...>  
wrote:
> All I have is something like:
>struct foo
> { PFC_MONO(foo) {PFC_VAR3(x, y, z);}
>   int x, y, z;
> };

That's admirably brief! I assume you're generating code rather than data  
though - which tends to be more verbose, or at least it does for me.

One trick I found helpful to avoid identifier repetition was to move some  
commas inside macros, to avoid having to specify the parent name twice in  
IMPLEMENT_BEGIN/IMPLEMENT_END and still avoid explict arity:

// rtti.h - detail elided
#define COMPOUND(name) static const member_t members[];
#define RTTI(name, members) const member_t name::members[] = { members {  
NULL, ... } };
#define MEMBER(name) { #name, ... },

// foo.h
struct foo
{
COMPOUND(foo)

int x, y, z;
};

// foo.cpp - essentially generates const member_t foo::members[] = { ... };
RTTI(foo, MEMBER(x) MEMBER(y) MEMBER(z))

Cheers,

Will

Re: [Algorithms] Complexity of new hardware

[Gregory J. <gj...@da...>]

Ours just looks like this (I'm sure this isn't unique to us):

class MyClass : public BaseObject
{
public:
	MyClass();
	~MyClass();

	HYP_CLASS(MyClass, BaseObject); // make this class part of
reflection
	HYP_CLASS_CREATABLE(); // lets you create one of these in the editor
	
	HYP_PROPERTY(Int32Property, "My property description, for editor", \
Int32, 12, 0); 	
	HYP_PROPERTY(StringProperty, "My property description, for editor",
\ 		String, "stringDef", 0);

	// and so on

private:
	// these don't partake in reflection
	int m_someMember;
	void* m_pSomeOtherMember;
};


No parsing or code generation needed. These properties insert a single
variable of the declared type in the class; you can access it directly via
the mangled name or through getters/setters (getInt32Property() for
example). The rest is done in macros and static init. It's primarily for the
convenience of data editing and marshaling from the editor output format;
there are no method calls in this system. It's worked well for us; once the
headache of putting together the property templates and macros is done,
there's nothing left but to use it.

But I agree with Jon, it would be nicer if there were something in C++ that
provided the same functionality in a first-class system; at this point I
really am just re-implementing the part of CLR reflection that we need.

Greg

Re: [Algorithms] Complexity of new hardware

[Jarkko L. <al...@gm...>]

Yes, amongst few other things PFC_MONO macro defines a template function for
introspection, and PFC_VAR() macros expand to calls to the property
enumeration interface passed to the function (e.g. PFC_VAR(x); ->
pe_.var("x", v_.x);) Because it's a function, you can also define custom
introspection which is sometimes necessary by writing code instead of using
PFC_VAR() macros.

The advantage of non-custom introspection is that it can perform automatic
data conversion when loading data which I have found extremely useful, i.e.
if you add/remove variables or change variable types it's still able to load
the old data. Otherwise you would have to hand-code the conversion which
could get quite a taunting task.


Cheers, Jarkko


-----Original Message-----
From: Will Vale [mailto:wi...@se...] 
Sent: Sunday, April 19, 2009 5:33 AM
To: Game Development Algorithms
Subject: Re: [Algorithms] Complexity of new hardware

On Sun, 19 Apr 2009 03:03:27 +1200, Jarkko Lempiainen <al...@gm...>  
wrote:
> All I have is something like:
>struct foo
> { PFC_MONO(foo) {PFC_VAR3(x, y, z);}
>   int x, y, z;
> };

That's admirably brief! I assume you're generating code rather than data  
though - which tends to be more verbose, or at least it does for me.

One trick I found helpful to avoid identifier repetition was to move some  
commas inside macros, to avoid having to specify the parent name twice in  
IMPLEMENT_BEGIN/IMPLEMENT_END and still avoid explict arity:

// rtti.h - detail elided
#define COMPOUND(name) static const member_t members[];
#define RTTI(name, members) const member_t name::members[] = { members {  
NULL, ... } };
#define MEMBER(name) { #name, ... },

// foo.h
struct foo
{
COMPOUND(foo)

int x, y, z;
};

// foo.cpp - essentially generates const member_t foo::members[] = { ... };
RTTI(foo, MEMBER(x) MEMBER(y) MEMBER(z))

Cheers,

Will


----------------------------------------------------------------------------
--
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
GDAlgorithms-list mailing list
GDA...@li...
https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Adrian S. <st...@8m...>]

This discussion has motivated me to finally write up an article on the
reflection system implemented in Day 1's Despair Engine.  Like Jarkko's, it
is implemented purely in C++ without the use of compiler extensions or a
custom preprocessor.

The article is pretty long, but if you're only interested in the actual
syntax you can jump straight to the bottom.  Enjoy!

http://gameangst.com/?p=107

Adrian

Re: [Algorithms] Complexity of new hardware

[Nicholas \Indy\ R. <ar...@gm...>]

I haven't used it, but additionally Insomniac has released their c++
reflection framework under a BSD style license along with their
Nocturnal initiative:
http://nocturnal.insomniacgames.com/index.php/Reflect.

Indy

On Sun, Apr 19, 2009 at 11:19 AM, Adrian Stone <st...@8m...> wrote:
>
> This discussion has motivated me to finally write up an article on the
> reflection system implemented in Day 1's Despair Engine.  Like Jarkko's, it
> is implemented purely in C++ without the use of compiler extensions or a
> custom preprocessor.
>
> The article is pretty long, but if you're only interested in the actual
> syntax you can jump straight to the bottom.  Enjoy!
>
> http://gameangst.com/?p=107
>
> Adrian
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>

Re: [Algorithms] Complexity of new hardware

[Tibor K. <tib...@gm...>]

I like how you managed to use the brackets and operators in such a way 
that it almost looks like normal code. Makes it much more readable than 
the usual macro voodoo :)

One things always leaves me wondering - do you guys actually do all your 
release build loading through the serialization interface or do you use 
a different approach there? I guess the former, since your net code 
depends on it. I always found these kinds of serialization approaches to 
be way too dynamic (with all the new/delete/smart pointer stuff) to be 
present in the release build where you want your memory to be as static 
as possible. Or is it just not a problem for you in practice?

Cheers,
Tibor

Adrian Stone wrote:
> This discussion has motivated me to finally write up an article on the
> reflection system implemented in Day 1's Despair Engine.  Like Jarkko's, it
> is implemented purely in C++ without the use of compiler extensions or a
> custom preprocessor.
>
> The article is pretty long, but if you're only interested in the actual
> syntax you can jump straight to the bottom.  Enjoy!
>
> http://gameangst.com/?p=107
>
> Adrian
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>
>   

Re: [Algorithms] Complexity of new hardware

[Jarkko L. <al...@gm...>]

Even when using serialization interface, you could for example store a total
memory required for a level/sector and allocate objects incrementally from a
single linear chunk of memory for it.


Cheers, Jarkko


-----Original Message-----
From: Tibor Klajnscek [mailto:tib...@gm...] 
Sent: Monday, April 20, 2009 12:58 AM
To: Game Development Algorithms
Subject: Re: [Algorithms] Complexity of new hardware

I like how you managed to use the brackets and operators in such a way 
that it almost looks like normal code. Makes it much more readable than 
the usual macro voodoo :)

One things always leaves me wondering - do you guys actually do all your 
release build loading through the serialization interface or do you use 
a different approach there? I guess the former, since your net code 
depends on it. I always found these kinds of serialization approaches to 
be way too dynamic (with all the new/delete/smart pointer stuff) to be 
present in the release build where you want your memory to be as static 
as possible. Or is it just not a problem for you in practice?

Cheers,
Tibor

Adrian Stone wrote:
> This discussion has motivated me to finally write up an article on the
> reflection system implemented in Day 1's Despair Engine.  Like Jarkko's,
it
> is implemented purely in C++ without the use of compiler extensions or a
> custom preprocessor.
>
> The article is pretty long, but if you're only interested in the actual
> syntax you can jump straight to the bottom.  Enjoy!
>
> http://gameangst.com/?p=107
>
> Adrian
>
>
>
----------------------------------------------------------------------------
--
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>
>   

----------------------------------------------------------------------------
--
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
GDAlgorithms-list mailing list
GDA...@li...
https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Adrian S. <st...@8m...>]

We do indeed do all of our final loading through reflection-driven
serialization.  The system doesn't require a lot of allocations though.  The
reflection code itself doesn't perform any memory management.  Objects must
be instantiated externally, and neither the serialization code nor the
underlying reflection code care if the addresses it deals with are on the
stack or part of some shared, pooled memory.

There is a cost to loading through the reflection system, however.  It comes
in the form of virtual function calls and small memory moves when memcpy
would suffice.  The biggest cost, I guess, comes from misuse at a higher
level.  It is reasonably efficient to serialize into and out of fields
represented by member variable pointers, but people have a tendency to want
to serialize into fields using non-trivial getters and setters.  Systems
like this that hide complexity also tend to obscure cost.  A hand-written
load function would never recompute cached state after loading each field of
an object, but a poorly written reflection definition that calls a custom
setter for each field might well end up doing that.  We try to encourage
people to think about how the reflection definitions they write will
translate into serialization code, but in the heat of development mistakes
are all too common.

Of course the bulk of our data is in resources like textures and meshes and
none of that is serialized through the reflection system.  Consequently even
though reflection-driven serialization is expensive, it has yet to be the
lowest hanging fruit in any of our profiles.

Adrian


-----Original Message-----
From: Tibor Klajnscek [mailto:tib...@gm...] 
Sent: Sunday, April 19, 2009 2:58 PM
To: Game Development Algorithms
Subject: Re: [Algorithms] Complexity of new hardware

I like how you managed to use the brackets and operators in such a way 
that it almost looks like normal code. Makes it much more readable than 
the usual macro voodoo :)

One things always leaves me wondering - do you guys actually do all your 
release build loading through the serialization interface or do you use 
a different approach there? I guess the former, since your net code 
depends on it. I always found these kinds of serialization approaches to 
be way too dynamic (with all the new/delete/smart pointer stuff) to be 
present in the release build where you want your memory to be as static 
as possible. Or is it just not a problem for you in practice?

Cheers,
Tibor

Adrian Stone wrote:
> This discussion has motivated me to finally write up an article on the
> reflection system implemented in Day 1's Despair Engine.  Like Jarkko's,
it
> is implemented purely in C++ without the use of compiler extensions or a
> custom preprocessor.
>
> The article is pretty long, but if you're only interested in the actual
> syntax you can jump straight to the bottom.  Enjoy!
>
> http://gameangst.com/?p=107
>
> Adrian
>
>
>
----------------------------------------------------------------------------
--
> Stay on top of everything new and different, both inside and 
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today. 
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>
>   

----------------------------------------------------------------------------
--
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
GDAlgorithms-list mailing list
GDA...@li...
https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Adrian wrote at 4/19/2009:
> http://gameangst.com/?p=107

The syntax approach is really interesting, but I find the overall
format too verbose. Still too much redundant data for my taste.

We use something much more terse:

meta ver(1) class ENGINE_API CDecalConfig : public CResource {
public:
  CDecalConfig(void);
  ~CDecalConfig(void);
  // ... functions ...
  
  MD_DECLARE;
public:
  _( 1) Vector2f dc_vSize; _("Size (m)")
  _( 2) IDENT dc_idTextureUVMap; _("Texture UV map")
  _( 3) IDENT dc_idShadowUVMap; _("Shadow UV map")
  _( 4) Ptr<CShaderPreset> dc_pspShader; _("Material") flags(SMF_NEW|SMF_DELETE|SMF_BROWSE)
};

This of course, requires a custom parser, but I think it is very much
worth the extra typing later on. I find all the approaches that force
you to repeat yourself to be dangerous. You can easily forget to list
one of the variables. With the preprocessor approach, in contrast, you
immediately see everything about a member when you look at its
declaration.

Also, using per-class C++ code that creates meta objects at boot time,
as you mention in the article, is really heavy on the final executable
size, and linker performance. We used that previously, but trying to
ship with that for Xbox1 forced us to rewrite it to use static data
per class, which is then read by the reflection code at boot time to
generate the needed objects. That approach makes handling templates
more complicated, though.

Alen

Re: [Algorithms] Complexity of new hardware

[Adrian S. <st...@8m...>]

The desire to keep the reflection definitions brief and part of the actual
class definition is actually contrary to our design goals for the system.

We anticipated very richly annotated reflection definitions and we didn't
want that information obscuring the class definitions.  This isn't a system
for publishing the memory layout of POD types to make writing objects to
disk and making endian conversion easier; it is framework for publishing
arbitrary metadata for C++ constructs.

To give you an idea, we have over 60,000 lines of reflection definitions in
our codebase and most of it is stuff I don't want to see when I'm reading
the header for a class.  A lot of it has to do with user interface design
and telling the cinema system which properties can be animated in what ways.
There is also stuff that doesn't tie directly to an actual member of a
class, like a fake "field" that reads data in an old format and writes
nothing--performing automatic version upgrade during serialization.

The redundant part of the system that I have the most mixed feelings about
is the manual specification of the name of a field.  Macros could have
automated that in most cases.  Having the name be explicit, however, has
proven useful on numerous occasions.  Our engineers like the flexibility to
refactor the code without having to worry about breaking serialization or
scripting.  It is perfectly valid to let a variable name in C++ diverge from
the string name in the reflection field.  It is also valid, and incredibly
useful, to remove a field entirely from a class and replace its
corresponding reflection entry with a getter/setter that emulates the old
field's behavior.

Adrian

-----Original Message-----
From: Alen Ladavac [mailto:ale...@cr...] 
Sent: Monday, April 20, 2009 6:16 AM
To: Adrian Stone
Cc: 'Game Development Algorithms'
Subject: Re: [Algorithms] Complexity of new hardware

Adrian wrote at 4/19/2009:
> http://gameangst.com/?p=107

The syntax approach is really interesting, but I find the overall
format too verbose. Still too much redundant data for my taste.

We use something much more terse:

meta ver(1) class ENGINE_API CDecalConfig : public CResource {
public:
  CDecalConfig(void);
  ~CDecalConfig(void);
  // ... functions ...
  
  MD_DECLARE;
public:
  _( 1) Vector2f dc_vSize; _("Size (m)")
  _( 2) IDENT dc_idTextureUVMap; _("Texture UV map")
  _( 3) IDENT dc_idShadowUVMap; _("Shadow UV map")
  _( 4) Ptr<CShaderPreset> dc_pspShader; _("Material")
flags(SMF_NEW|SMF_DELETE|SMF_BROWSE)
};

This of course, requires a custom parser, but I think it is very much
worth the extra typing later on. I find all the approaches that force
you to repeat yourself to be dangerous. You can easily forget to list
one of the variables. With the preprocessor approach, in contrast, you
immediately see everything about a member when you look at its
declaration.

Also, using per-class C++ code that creates meta objects at boot time,
as you mention in the article, is really heavy on the final executable
size, and linker performance. We used that previously, but trying to
ship with that for Xbox1 forced us to rewrite it to use static data
per class, which is then read by the reflection code at boot time to
generate the needed objects. That approach makes handling templates
more complicated, though.

Alen

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Monday, April 20, 2009, 8:45:23 PM, Adrian wrote:
> To give you an idea, we have over 60,000 lines of reflection definitions in
> our codebase and most of it is stuff I don't want to see when I'm reading
> the header for a class.

Then we have to agree to disagree. I want to know everything about a
member when I see its declaration. You obviously don't. Well, I
wouldn't want to either, if it was that verbose. It really would
clutter the header. But I argue that we can pull all the meta info
needed in a very light-to-read way. I don't think it is the info
itself that creates the verboseness. It is the requirement to have the
problem solved by the compiler. The trick is that this is orthogonal
to portability, and you don't need the compiler at that stage.

> A lot of it has to do with user interface design and telling the
> cinema system which properties can be animated in what ways.

Similar stuff here. UI-specific info, network-specific info, hints on
how to behave on clipboard operations, per-class non-member info, any
additional custom info etc. We also use the system to define
meta-functions that can be called on a generic object, bindings for
scripting, bindings to global variables (cvars), global functions (for
scripting), etc.

And it is all very terse. Because we feel we need terseness to be
quick and more error-proof (re once-and-only-once) when writing new
code. I understand you don't care about that as much.

> There is also stuff that doesn't tie directly to an actual member of
> a class, like a fake "field" that reads data in an old format and
> writes nothing--performing automatic version upgrade during
> serialization.

Isn't that unnecessary? With proper format version tracking, you can
delete and add members to a class without keeping the old members as
clutter (in 99% most cases).

> It is perfectly valid to let a variable name in C++ diverge from the
> string name in the reflection field.

We support that as an option, but it gets a bit awkward, and prone to
search-replace errors. I prefer the number + name + "editor name"
better.

JM2C,
Alen

Re: [Algorithms] Complexity of new hardware

[Adrian S. <st...@8m...>]

> From: Alen Ladavac [mailto:ale...@cr...]
> Similar stuff here. UI-specific info, network-specific info, hints on
> how to behave on clipboard operations, per-class non-member info, any
> additional custom info etc.

Would you mind sharing a field definition that contains that level of
annotation?  You can see how cluttered our definitions get in my article,
but I'm having a hard time envisioning it in your syntax.

> Isn't that unnecessary? With proper format version tracking, you can
> delete and add members to a class without keeping the old members as
> clutter (in 99% most cases).

The situation I'm trying to describe is this: someone writes a class that
holds an angle in degrees.  Later they decide it would be more prudent to
store the value in radians.  They aren't just removing a field, they're
converting it.  In our system the original reflection definition would look
like this:

.field("angle", &MyClass:m_angle)
[
	Prop::SaveLoad()
]

And the new definition would look like this:

.field("angle", &MyClass::GetAngleDegrees, &MyClass::SetAngleDegrees)
[
	Prop::Load()
]
.field("angleRadians", &MyClass::m_angle)
[
	Prop::SaveLoad()
]

Other than the unfortunate code bloat of adding the getters and setter for
degrees, the reflection definition automatically converts old data and loads
new data with zero cost.  It also preserves compatibility with scripts.
When we can afford to stall production and batch process all the data for
all the projects we have going, we can remove the "angle" reflection field
from the code.

Adrian

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Monday, April 20, 2009, 10:13:12 PM, Adrian wrote:
> Would you mind sharing a field definition that contains that level of
> annotation?  You can see how cluttered our definitions get in my article,
> but I'm having a hard time envisioning it in your syntax.

Here are some examples (from different classes):

  _(1) Ptr<CFlareConfiguration> ep_pfcConfig; _("Flare")  flags(SMF_NONE|SMF_BROWSE|SMF_NEW)
 ....
  _(3) CEditMesh         *msh_pemEdit;       flags(SMF_OWNED|SMF_EDIT|SMF_CLIPBOARDBARRIER)
 ....
  _(6) FLOAT pwp_fMinimumWoundEffectStrength; _("Minimum wound effect strength" min="0.0f" max="1.0f")
...
 _(211) net QuatVect en_qvNetPlacement; context(CTX_ABS_PLACEMENT)
 _(212) net Vector3f en_vNetVelocity; context(CTX_VELOCITY)
...
  meta unreliable compacted client virtual void SetDriveSteerRatioAndLookDir(Vector3f vDriveSteer, Vector3f vLookDirEul);
...
  meta saved cvar ENGINE_API extern FLOAT efx_fFixedEV; _(renoption perdocument version="2")
...
  meta cvar ENGINE_API extern BOOL ren_bDynamicShadows; _("D&ynamic Shadows" renoption)

Note that most members need only some part of the specs, because the
assumed defaults are set to the common denominator. Also, as you can
see, there are different types of tags ranging from keywords, through
preset flags, to completely flexible hint strings.

> The situation I'm trying to describe is this: someone writes a class that
> holds an angle in degrees.  Later they decide it would be more prudent to
> store the value in radians.  They aren't just removing a field, they're
> converting it.

That's an interesting approach. We have very rarely observed this in
practice, so when it happens, we just keep the old field, hide it from
user, and convert in PostRead() meta-function. This means some extra
memory and disk space until we can trim, as you noted below. We could
save disk space by tagging it as load-only-no-saving. But I do like
the idea that this could be removed from memory immediately.

IME for serialization compatibility in 99% cases you need member
removing/adding/retyping members, and changing base classes. So I
stick to the adage: "Simple things should be simple, complex things
should be possible", and in that spirit don't want to write long
declarations for all members just because 1% will need some special
handling. YMMV.

> .field("angle", &MyClass:m_angle)
> [
>         Prop::SaveLoad()
> ]

This reminds me of one thing I was meaning to comment earlier... I
prefer opposite logic to this. If I don't specify anything, I'd
assume save+load would be the default as that's most often used. But
again, some like it terse, some don't. :)

Alen

Re: [Algorithms] Complexity of new hardware

[Adrian S. <st...@8m...>]

> From: Alen Ladavac [mailto:ale...@cr...]
>   _(6) FLOAT pwp_fMinimumWoundEffectStrength; _("Minimum wound effect
> strength" min="0.0f" max="1.0f")

Thanks!  Very interesting.  Definitely just a different aesthetic taste
here.


> IME for serialization compatibility in 99% cases you need member
> removing/adding/retyping members, and changing base classes.

Not sure if this was clear, but we don't need the multiple field shenanigans
in the simple cases either.  Removing a field without preserving the data is
just a matter of deleting it.


> This reminds me of one thing I was meaning to comment earlier... I
> prefer opposite logic to this. If I don't specify anything, I'd
> assume save+load would be the default as that's most often used. But
> again, some like it terse, some don't. :)

I tend to agree with you, but some of my coworkers made a compelling
argument in the other direction.  They argued that forgetting to serialize a
field was an obvious mistake that would be caught immediately, whereas
forgetting to NOT serialize a field is a silent error that causes no harm
other than inefficiency and consequently will never be caught.

Adrian

Re: [Algorithms] Complexity of new hardware

[Alen L. <ale...@cr...>]

Adrian wrote at 4/20/2009:
>> IME for serialization compatibility in 99% cases you need member
>> removing/adding/retyping members, and changing base classes.
> Not sure if this was clear, but we don't need the multiple field shenanigans
> in the simple cases either.  Removing a field without preserving the data is
> just a matter of deleting it.
Yes, I understood it that way. The comment was aimed at
"[Prop::SaveLoad()]", but I covered that better later, so it has
become redundant.

> forgetting to NOT serialize a field is a silent error that causes no harm
> other than inefficiency and consequently will never be caught.
I wouldn't consider it a potential to cause any significant
inefficiency. In our codebase the temporary (non-serialized) fields
are <1% in declarations and even way less in byte count. Btw, there is
a large amount of classes that carry reflection info just for
convenience of using it for dynamic type handling in the UI (windows,
dialogs, etc.), but they are not serialized anyway, so we don't care
what's the serialization flag on their members. IME, the largest
percentage of file size (across all files in a project) is taken by
POD arrays like textures, vertex arrays, etc. YMMV.

Alen

Re: [Algorithms] Complexity of new hardware

[Mike S. <mik...@gm...>]

On Sat, Apr 18, 2009 at 10:27 AM, Jon Watte <jw...@gm...> wrote:
> Yes; the only thing that is annoying is the repeating-yourself problem.
> I find I usually need to repeat myself one and a half time.
> First, it's the declaration.
> Then, it's the repeat of the declaration, in macro form.
> Finally, it's some small bits of macro that builds the
> helpers/visitors/registration needed to make it all "go."

With a full C++ parser available, you could generate a lot of that
from just the declaration, right?

https://developer.mozilla.org/en/Dehydra

Requires a GCC plugin, so the code you want to scan has to be
compilable by GCC, but that's not a very large burden considering the
benefits you get.  We use Dehydra analyses to ensure that certain
classes are only ever allocated on the stack, and to ensure that
certain transitive invariants hold (like locks-not-held, etc.) with
simple annotations in the source code.

Mike

Re: [Algorithms] Complexity of new hardware

[Adrian B. <ad...@gm...>]

Hand coding incremental conversions is really easy.  Seems to me the
problem is how to flush out old versions so you don't end up with a
rats nest of compatibility code.  Fortunately, it's only a matter of
writing other code :).

Combining an easily editable source format (e.g. text) for your assets
with a translation application allows these sorts of rebuilds easily.
Layering incremental versioning on top is not hard.  There are
certainly trade offs: mergeability, better abstraction in some cases,
more code, more time to get to preview, etc.

Or, if you want to keep your source in a compiled format, providing
some form of mass translation mechanism should work.  E.g.
snapshotting version changes into a file which can be run across all
assets.  I haven't done this personally, and it comes with a different
sort of baggage, but seems doable all the same.

Pick your poison, but there are alternatives to brain dead simple
serialization that have better long term usability.  Also, unless you
have a baking step (hello translation), it feels like this would be
important for loading performance anyways.

Cheers,
Adrian

On Sun, Apr 19, 2009 at 3:17 AM, Jarkko Lempiainen <al...@gm...> wrote:
> Otherwise you would have to hand-code the conversion which
> could get quite a taunting task.

Re: [Algorithms] Complexity of new hardware

[Adrian B. <ad...@gm...>]

If you can break off a script by diffing two sets of type metadata is
that hard coded?  It would be unnecessary maintenance work, unless you
want to be able to change your data and can't do it within your
framework :P.  Maybe there's a way I haven't thought of.  Every time I
look at C# serialization, I've come up with a number of things I
wouldn't be able to do very easily (rename B, break A into 2 pieces,
etc.).

I certainly hadn't thought about it from a middleware perspective, but
all middleware needs a data upgrade path between versions.  Otherwise
you get stuck in time and become another point in favor of rolling
one's own tech.

Unless I'm missing something, you can't strip off all the class/type
information unless all your data is in the final format.  At that
point, you've solved your upgrade problem.  Linearly allocating
_definitely_ helps.  For some, though, even running simple
serialization code may not be fast enough.

Cheers,
Adrian

On Sun, Apr 19, 2009 at 3:53 PM, Jarkko Lempiainen <al...@gm...> wrote:
> Hand coding data coversion just sounds awful lot of unnecessary maintenance
> work which I rather avoid when constantly adding/removing variables to/from
> classes. Batch processing data for data conversion code clean-up also
> assumes you got all the old data available for processing, which may cause
> problems particularly if you are a middleware developer.
>
> For improving loading performance and reducing data size I can strip off all
> the type information data for final build, in which case loading becomes
> straight forward linear data streaming.
>
>
> Cheers, Jarkko
>
>
> -----Original Message-----
> From: Adrian Bentley [mailto:ad...@gm...]
> Sent: Monday, April 20, 2009 12:30 AM
> To: Game Development Algorithms
> Subject: Re: [Algorithms] Complexity of new hardware
>
> Hand coding incremental conversions is really easy.  Seems to me the
> problem is how to flush out old versions so you don't end up with a
> rats nest of compatibility code.  Fortunately, it's only a matter of
> writing other code :).
>
> Combining an easily editable source format (e.g. text) for your assets
> with a translation application allows these sorts of rebuilds easily.
> Layering incremental versioning on top is not hard.  There are
> certainly trade offs: mergeability, better abstraction in some cases,
> more code, more time to get to preview, etc.
>
> Or, if you want to keep your source in a compiled format, providing
> some form of mass translation mechanism should work.  E.g.
> snapshotting version changes into a file which can be run across all
> assets.  I haven't done this personally, and it comes with a different
> sort of baggage, but seems doable all the same.
>
> Pick your poison, but there are alternatives to brain dead simple
> serialization that have better long term usability.  Also, unless you
> have a baking step (hello translation), it feels like this would be
> important for loading performance anyways.
>
> Cheers,
> Adrian
>
> On Sun, Apr 19, 2009 at 3:17 AM, Jarkko Lempiainen <al...@gm...>
> wrote:
>> Otherwise you would have to hand-code the conversion which
>> could get quite a taunting task.
>
> ----------------------------------------------------------------------------
> --
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> GDAlgorithms-list mailing list
> GDA...@li...
> https://lists.sourceforge.net/lists/listinfo/gdalgorithms-list
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_name=gdalgorithms-list
>