Re: Ideas on the future of Quesa.

[Roger H. <rog...@mi...>]

Thank you Dair, a very BIG thank you. I hope we can manage without you.


> - Don't lose abstraction unless you really need it. I'm quite pleased
> with the way the internals turned out, in that we kept a very OOish
> style internally (where each .c file exposes some opaque types and
> functions to work on those types; so geometry code doesn't see any
> details of cameras, etc).

The reason for my proposal to use C++ in Quesa is that we can use
the 'private' keyword to keep the internal details of a class private
whilst allowing us to have it in the e3xxx.h file and subclass from it,
which gives us better efficiency.

If that is as far as we can agree to go then that is fine with me, but
I think there are other benefits of C++ which we could usefully use.
I am not advocating the gratuitous use of C++'s more exotic features,
just where there are real benefits in terms of either efficiency or
readability.

> - Keep the plug-in renderer idea. This is one of the nicest things 
> about
> QD3D/Quesa IMO, the idea that the organisation side of things is
> completely separate from the rendering side.

Absolutely.

> - Think about changing data structures. The public QD3D types have
> several limitations; vectors/colours are triples rather than quads (so
> not so amenable for AltiVec processing),

Yes, now the G3 is almost history we should be thinking seriously about
an AltiVec version of Quesa. 3DVector type should have a 0.0 as its 
fourth
element and 3DPoint should have 1.0 as the fourth element. Should
colours have the alpha as the fourth element?


>  and the structures don't have size fields in them.

> One of the problems with the QD3D API is that, like a C++ API, it's not
> good for backwards compatibility. Not saying that MS has all the
> answers, but the Win32 style of having a size field at the start of 
> each
> structure at least gives you the option of adding new fields without
> breaking older apps (although nowadays the Carbon approach of opaque
> types and generic accessor functions is probably better).

So are we talking about accessor functions for 3DPoint etc? That seems a
bit over the top to me. Quesa's object are already opaque. I know we do
tend to store the Q3xxxdata structures in them but that is up to us to 
change
if we need to change the internal representation we can do so easily. I 
don't
understand this, could someone explain?

> - Look at GL/D3D. The way data (texture and geometry) is submitted to
> cards has changed quite a bit since the RAVE time frame, and the key to
> getting good performance nowadays is to minimise the interaction the 
> CPU
> has with the data and submit data in as large a batch as possible.

> Ideally you just want to have the data in memory somewhere, give the
> card a pointer to it, and let it handle the transfer itself (with a VBO
> style lock/unlock to synchronise access).
>
> - Look at some other scene graphs. We kept ourselves focused on just
> doing QD3D initially, which I think was a good plan (as it meant things
> didn't go off in a hundred different directions), but there are other
> libraries out there which have good ideas.
>
> Wouldn't hurt to take a look at some of them, as exposing something 
> like
> shading languages will need to be done in a similar way (no point in
> inventing our own shading language, but we should have something that
> lets you plug in either ARB or D3D shaders).

I know very little about these aspects of Quesa so will leave it to 
others to comment.

> - Keep the history. Even if you move on to using subversion at some
> point, I would recommend keeping as much of the CVS history as 
> possible.

Yes.

>
> We don't go all the way back to 1999, which is something I always
> regretted, but I think it's useful to be able always get back to older
> versions. If some planned development doesn't work out, that means you
> can then rewind N months and easily start again.
>
> - Keep the licence. Switching from the LGPL to BSD solved a lot of
> problems for me at work, and allowed me to use the math sections of
> Quesa in our D3D->GL translation library. The GPL licences wouldn't
> have.

Yes.

>
> - Think about a CoreFoundation approach. Something that's worked well
> for Apple is to have a CoreFoundation library which provides basic
> primitives (strings, sets, etc) and then have other libraries built on
> top.
>
> Something similar could help popularise the library and make it easier
> for people to dip into. E.g., regular requests on the mac-opengl and
> mac-games-dev lists are requests for file format readers and 3D math
> code. So:
>
> - 3D math library. All the standard operations, intersections, etc; you
> could see this growing into more general purpose fields like collision
> detection, visibility tests, or physics.
>
> - 3D file format parser (harness that would register N plug-ins for
> doing IO, the plug-ins parse their data and return it back via 
> callbacks
> with some type info.
>
> Means that people could link to the file format library, and get called
> back as objects are parsed out of the file; the object description
> allows them to build their own data structure out of it. Quesa would
> also link to the file format library, and have a set of callbacks that
> build Quesa objects out of the data.
>
> I know we moved the Viewer library into Quesa so we could have a single
> shared library for users, but think of these more like the plug-in
> renderers.
>
> I.e., a well defined interface between them and the rest of Quesa, and
> build them (and perhaps even the plug-in renderers) as static libraries
> that then get linked into the Quesa shared library.
>
> For this to work the interface would need to be 100% independent; i.e.,
> you would need to be able to say "I just want to use the math library"
> and know exactly what headers/.lib files were needed for your app (and
> the math library couldn't call anything in Quesa; it would have to sit
> under everything - perhaps need a CF-style library to hold truly common
> bits below these utility libraries).

For the future, will try to bear them in mind.

>
> - Split up the source files. Some of them (math, camera, etc) are
> getting too large and could benefit from what happened to E3Geometry.c.
>
> I.e., have an E3Camera.c for the common stuff; then put the individual
> camera types into their own E3CameraFoo.c files. If you're thinking
> about C++ this would be a good first step, as it means you then have 
> one
> C++ object per source file and keep everything well separated.

Or maybe a few very simple objects.

>
> - Don't go crazy. :-) One of the problems with C++ IMO is that it's 
> very
> easy to create systems that are just unmaintainable. Everyone feels
> differently about what's good and what's bad about C++, but I've seen
> plenty of projects that just got out of control when they started 
> taking
> a "C++ purist" view of things rather than the pragmatic view.

I agree entirely.

>
> The STL is definitely a useful source for all the standard data types
> (lists, vectors, hash tables, etc), but I would be wary of going
> overboard with stuff like operator overloading.
>
> One of the major problems on the games we see on the games we port is
> that CW/gcc do a terrible job of keeping simple C++ objects like a
> vector tuple in registers - so overloading "+" to give you a vector
> addition for "x + y" will often generate truly dire code compared to
> adding the fields yourself.

But the code is so much more readable, computers are getting faster
all the time and compilers are becoming more efficient. Microspot
could not maintain big chunks of 3DWorld without operator overloading.
A single line of code would have to become a dozen statements with
temporary variables, awkwardly named procedure calls with & operators
spread about all over the place. As you say, 'everyone feels differently
about what's good and what's bad about C++'. How do you other
guys think about operator overloading for vectors, points and matrices?

> Similarly for exceptions - I've yet to see a project where they helped,
> but have seen several where they were the beginning of the end. :-)

I am very happy to keep exceptions either out of Quesa or entirely
contained within it. We (Microspot) have wrappers for all of the
Quesa routines which throw exceptions on an error. That way we
can use the function result for what its intended for - the result of
the calculation and not for an incidental fact that there was not an
error. I would soon tire of writing code with every line being an
if statement testing for errors, but I guess that's what the rest of
you have been happy with for the last few years.

> But I'm sure you guys have your own ideas, so feel free to take the
> project in whatever way makes sense for you.

Thanks.

> I've had a lot of fun working on Quesa, and definitely learned a lot
> about what works and what doesn't (both 3D and otherwise).
>
> I'll stay on the list for a day or so, but will be unsubscribing later
> this week - my email will be dair AT refnum.com for the forseeable
> future, so feel free to drop me a line if you want my 2 pence on
> anything. :-)
>
> But for now, have fun - and good luck!

Thanks, we will try.




On Monday, November 29, 2004, at 10:42  pm, Jose' Cruanyes wrote:

> Given that the external API doesn't change...

Of course, but the API is large there may be parts of it which nobody 
actually uses.

> I've to second James on this, and my proposal is:
>
> + C++,
> 	I think averybody agree
> + few STL and few exceptions,
> 	containers, iterators and functors when it makes sense, not to 
> templatize everyting,
> 	exceptions inside Quesa

So I think this boils down to using these when they are in a standard 
library, and not define any of our own.
That sounds a sensible approach. Of course we can still discuss 
breaking this rule if everyone agrees in
a particular instance, but none of us will do so without approval from 
the others. Nearly every use of STL
will need to be wrapped in a try{}catch(...) as a minimum.


> + inheritance
> 	yes the class hierarchy is best described by inheritance, BUT no 
> virtuals functions to implement the API, instead I'll make an array of 
> function pointers in the class with a slot for every kind of api 
> function ( a shortcut to the metahandler) and a hashtable  to plugin 
> methods
>
> so
> instance.tablePtr = &class.functionArray[];
> instance.hashtablePtr = &class.hashtable;
>
> the functionArray is initialized by the class constructor, so it's 
> overriden while the class object is constructed
>
> creating the virtual methods with a custom method table, allows the 
> creation with plug-ins via the plug-in metahandler
>
> this way allows us also a mild transition. create the C++ base classes 
> and derive the others via the metahandler, when things works, begin 
> changing the implementation of derived classes in C++.

This is certainly the first thing to do, and is my main reason for 
proposing it. As for
virtual functions see below.


On Monday, November 29, 2004, at 06:53  pm, James W. Walker wrote:

>> But as a straw pole are we in favor of a real C++ implementation 
>> using inheritance, virtual methods but with no exceptions to replace 
>> the current C based E3 files. Yes or No.
>
> If I had to say just Yes or No on that, I guess I'd have to say No. I 
> think it might be better to use exceptions (and constructors and 
> destructors) than to use inheritance and virtual methods.  As I said 
> in another message, you can't really use STL without using exceptions 
> to some degree.
>
> I'd like to hear more about how you envision using inheritance and 
> virtual methods.  It's not clear to me how we could use inheritance 
> for the Quesa object hierarchy without losing the ability to extend it 
> with plugins, but maybe that's not what you had in mind.


Ok so I was trying to gather your comments on virtual functions and you
seem to be dead set against them, but so was I at first. If you are 
still
against them after reading this I will drop the idea. 0xDEADD0D0.

But first inheritance. The instance data of a Quesa sub class is ideally
held in memory just after the base class's data. That way we can 
allocate
just one block of data for all the instance data of a leaf class and 
all its
base classes in one go, and we do not have to follow a chain of pointers
to find a particular sub class's instance data. This has shown a great
performance boost for processing and rendering of complex documents.
Of course we could do this with structs, by defining a first data member
to be the base class's instance data, but we do want to keep the data
private to a class and not have it messed about with by derived classes,
so to have a struct/class inherit all the data members from its base 
class
privately is the only way to do this properly. Of course this also 
inherits
the methods but if we don't want this then we just don't define any 
methods.

Virtual methods. The other big speed up I made in Quesa was by
implementing in C the closest thing I could to a C++ VTable. I could
bring this code forward into cvs Quesa but the thought of allowing the
C++ compiler to do it for me is appealing as it is simper, less prone to
error and more readable. Where there is now a hash table look up for
a method we would instead use a virtual method. This would not of
course affect the API, just as the API does not expose the existing
functionality. There is of course the problem of extensibility. It might
be possible to have an extension class which would get called for
extension classes and would then vector off in the existing way.
However, apart from renderers, how many of us have actually
extended Quesa, I have but I would be very happy to modify my
extension into clean C++ classes linked into Quesa instead of the
horrid mess of meta handlers and multiple functions which
were necessary with the closed source library QD3D.

So have I talked you around?

Roger.