Re: [opendemo-devel] file format

[Dr. U. G. <Uwe...@ph...>]

Hello!

> I think we should consider changing the file format as follows

> to speed up playback,
Sounds good.

> use much less memory,
Do we have a problem here? Has the QVM a memory restriction?

> and simplify things:
I'm always for the most simplified solution.

> - Record only one map per xml file, with each xml possilbly including a
> reference to the next demo.
Was it ever possible to record more than one map? It is possible to play more
than one map in Q1 and QW and Q2 but all important movies I'm aware of, use
this reference technique to simplify demo editing. Even games like Half-Life,
where one level is really small have to reload the next level. If the demo file
changes as this time too is not so important.

> We really need to do this if we want to allow
> rewinding of demos during playback, since we can't store demos in memory. I
> don't see how this hurts functionality.
So you think about using odseek() to get to a special moment in the past?

> - Get rid of the 16 states used for delta encoding, and force delta'ing only
> from the previous snapshot (or baselines). This reduces the size of the od
> struct by about 9 megs, and should completely remove the odpStartElement
> performance bottleneck. We can keep the "currentSeq" and "deltaSeq" xml
> attributes, just in case.
That is reasonable. We can't miss a single frame, we are the server after all,
so we can delta encode as we like it best. I never understood which one of
delta encoding against the previous snapshot or baselines is any better. How
can the recording mechanism decide this?

> - Maybe move element data into attributes? E.g instead of:
>     <currentOrigin index="0">586.87023</currentOrigin>
>     we could use:
>     <currentOrigin index="0" value="586.87023" />
> I say this because AFAIK there is less variation allowed for attributes than
> character data. For example, I think leading and trailing whitespace must
> always be removed from attributes,
That's true only for non-CDATA attributes.
As short look into our DTD reveals, we use enumeration type (0|1|2) or NMTOKEN.
With the production rules from XML (here S denotes white space)
[3] S        ::= (#x20 | #x9 | #xD | #xA)+
[4] NameChar ::= Letter | Digit | '.' | '-' | '_' | ':' |
                 CombiningChar | Extender
[5] Name     ::= (Letter | '_' | ':') (NameChar)*
[6] Names    ::= Name (S Name)*
[7] Nmtoken  ::= (NameChar)+
[8] Nmtokens ::= Nmtoken (S Nmtoken)*
we see clearly, that you are right. There are no white-spaces at both ends.
So I would even think, leading and trailing attribute white spaces let the XML
file in the state "well-formed" but it is no longer conformant with our DTD.

Just for completeness, I choose NMTOKEN because it has some kind of restriction
over CDATA but there is no "floating point" attribute type, which would have
been better.

> while character data may or may not need
> that whitespace (depending on the DTD).
And the attribue xml:space.

> Since we don't use a validating XML
> parser to handle these variances, maybe we should move to attributes?
If it simplifies something, go for it.

I'll write a XSL stylesheet to convert old files into new ones (if we finally
decide on the new format). That is much better than supporting both formats or
writing a dedicated application for this task. There are so many free XSLT
processors available (http://www.xmlsoftware.com/xslt/), it should be no
problem to get one of them running on every computer system.

But we should never forget, that we hope to reuse much of all this framework
for other games as well. Maybe the names of the entity parameters are called
a bit different but all this XML file format stuff and delta-encoding technique
must be reusable at least for RTCW and maybe also also for DOOM3.

Bye, Uwe