opendemo-devel

[opendemo-devel] AddFloat corrected

[Dr. U. G. <Uwe...@ph...>]

Hello!

AddFloat() contained one of the first problems I ever solved for OpenDemo:
A type conversion with LCC (for bytecode) like
float f;
int i;
i = (int)f;
rounds the value sometimes up and does not always round down like GCC and
most other compilers do.
I had a correction for this in bg_lib.c but it somehow disappeared.

Now it works again and two servers with bytecode or Linux native libraries
(without any logged in clients, simply dedicated server of q3dm0) generate
the exact identical recording file. Only configstring index="0" is different
(ordering problem, I'll look into it).

Bye, Uwe

Re: [opendemo-devel] file format question

[Conor D. <co...@ma...>]

Hello,

> > I don't know how an entity can contain such characters, since it would
only
> > contain integer and floating point data,
> They are binary data embedded into plain text. Why shouldn't the binary
> representation of a floating point number contain a '<'? I'm speaking
> here of this odFprintf() call in odrWritePlayer() and odrWriteEntity(),
where
> a binary buffer will be pasted unchanged into the text XML file.

Unless I screwed something up, the "binary buffer" you speak of should only
contain plain text (it was filled in by odrDelta). The problem areas would
be in odrAppendConfigstring and odrAppendServerCommand, and possibly others.

> > Fixing the parser should be easy enough, where it calls the
SAX->characters
> > for the preceding raw data, then makes a 1-length string containing the
> > escaped character and calls SAX->characters for it, then repeats until
the
> > character data is exhausted.
> BTW, from which SAX parser is all his code copied? Or is all this
self-made?

I took the basic function layout and error codes from GNOME's libxml, and
coded the bodies of the functions myself.

> > Fixing the writer would be a little more work, but shouldn't be too
hard. I
> > think it would take a mbXMLSprintf or something, where it scans the
given
> > string for forbidden characters and escapes them properly when writing
to
> > the memory buffer.
> We have only 3 "bad" characters:
> '<' -> &lt;
> '>' -> &gt;
> '&' -> &amp;
>
> And the parser should restore the original character.

Agreed.

-Conor

Re: [opendemo-devel] file format question

[Dr. U. G. <Uwe...@ph...>]

Hello!

On Wed, Sep 19, 2001 at 11:52:03PM -0500, Conor Davis wrote:
> I don't know how an entity can contain such characters, since it would only
> contain integer and floating point data,
They are binary data embedded into plain text. Why shouldn't the binary
representation of a floating point number contain a '<'? I'm speaking
here of this odFprintf() call in odrWritePlayer() and odrWriteEntity(), where
a binary buffer will be pasted unchanged into the text XML file.

> Fixing the parser should be easy enough, where it calls the SAX->characters
> for the preceding raw data, then makes a 1-length string containing the
> escaped character and calls SAX->characters for it, then repeats until the
> character data is exhausted.
BTW, from which SAX parser is all his code copied? Or is all this self-made?

> Fixing the writer would be a little more work, but shouldn't be too hard. I
> think it would take a mbXMLSprintf or something, where it scans the given
> string for forbidden characters and escapes them properly when writing to
> the memory buffer.
We have only 3 "bad" characters:
'<' -> &lt;
'>' -> &gt;
'&' -> &amp;

And the parser should restore the original character.

Bye, Uwe

Re: [opendemo-devel] make system

[Dr. U. G. <Uwe...@ph...>]

Hello!

On Thu, Sep 20, 2001 at 12:05:27AM -0500, Conor Davis wrote:
> That's what I had tried to do initially with my makefiles, and it was a big
> reason why I got frustrated and gave up. I envisioned a qvm.mak, debug.mak,
> optimize.mak and such that would specify the compiler, flags, and file
> extensions. The Makefile would include one of these and specify the files to
> be compiled and the targets.
That is too difficult. The Autoconf/Automake system teaches us, that it is
much easier to do everything in one Makefile and you give your target on the
commandline.
Good targets would be
	linux
	linux-debug
	win32
	win32-debug
	bytecode
And additional shortcuts like
all: linux win32 bytecode
debug: linux-debug win32-debug
The main thing is to make sure, that the resulting compiling commands would
also run on win32 platforms, so rm->del and similar changes should be done
automatically. Fortunately, windows understands / as the PATH separator in
commands. But in every case, something like the initial command "config"
must be done under Windows too and Perl (for wpu) is mandantory as well.
We could even reduce all external program calls into mini-perl scripts like
in this Makefile snippet:

RM=perl -e 'unlink @ARGV;'

clean:
	$(RM) *.o

So we rely only on one external program (Perl), which everyone should have
anyway.

I'm currently trying to compile a Win32 DLL with my Win32-cross-compiler under
Linux. Then the binary package could contain a windows native library too.

> The tricky part here is getting gcc and the
> bytecode compiler (was a linux-native bytecode compiler ever released?)
I would think, that opendemo/bin/q3a/ contains a perfectly native Linux (ELF)
bytecode compiler (beside the old native Windows .exe files).

> to
> behave the same, and knowing when to compile g_syscalls.c (native mode) or
> just link g_syscalls.asm directly (qvm mode).
That is a bit tricky but not too much.

The main benefit of a standard Makefile would be the automatic dependency
calculation via "gcc -M".

> I don't want to abandon the Windows batch files, because newbie coders would
> be reliant on them. The linux Makefiles should have enough info to generate
> them automatically.
Yes, thats a good idea. They are not really necessary for "professional
builders" but can be generated (and packed into the source package files).

Uwe

Re: [opendemo-devel] make system

[Conor D. <co...@ma...>]

That's what I had tried to do initially with my makefiles, and it was a big
reason why I got frustrated and gave up. I envisioned a qvm.mak, debug.mak,
optimize.mak and such that would specify the compiler, flags, and file
extensions. The Makefile would include one of these and specify the files to
be compiled and the targets. The tricky part here is getting gcc and the
bytecode compiler (was a linux-native bytecode compiler ever released?) to
behave the same, and knowing when to compile g_syscalls.c (native mode) or
just link g_syscalls.asm directly (qvm mode).

I don't want to abandon the Windows batch files, because newbie coders would
be reliant on them. The linux Makefiles should have enough info to generate
them automatically.

-Conor

----- Original Message -----
From: "Dr. Uwe Girlich" <Uwe...@ph...>
To: "OpenDemo Development List" <ope...@li...>
Sent: Wednesday, September 19, 2001 1:58 AM
Subject: [opendemo-devel] make system


> Hello!
>
> Shouldn't we try to reorganize the make system? I hate it to have a
Makefile
> for Linux native, a shell-script for bytecode compiled under Linux, a
> batch file for bytecode compiled under Windows and a project file for
native
> Windows code. Any change of source file names must be reflected in in all
these
> places! Woudn't it be much better to have a standard Makefile with
different
> sets for compiler and compiler flags.
> And BTW, changes in header files don't trigger recompiles of any source
files,
> where these header files are included in the current system.
>
> Bye, Uwe
>
>
> _______________________________________________
> opendemo-devel mailing list
> ope...@li...
> https://lists.sourceforge.net/lists/listinfo/opendemo-devel
>

Re: [opendemo-devel] file format question

[Conor D. <co...@ma...>]

I don't know how an entity can contain such characters, since it would only
contain integer and floating point data, but as of right now my XML
writer/parser can't handle any forbidden characters in string data such as
configstrings and server commands. This is a real problem, because the game
vm could print a message containing a '<' or such, and the resulting XML
would not be well-formed.

Fixing the parser should be easy enough, where it calls the SAX->characters
for the preceding raw data, then makes a 1-length string containing the
escaped character and calls SAX->characters for it, then repeats until the
character data is exhausted.

Fixing the writer would be a little more work, but shouldn't be too hard. I
think it would take a mbXMLSprintf or something, where it scans the given
string for forbidden characters and escapes them properly when writing to
the memory buffer.

-Conor

----- Original Message -----
From: "Dr. Uwe Girlich" <Uwe...@ph...>
To: "OpenDemo Development List" <ope...@li...>
Sent: Wednesday, September 19, 2001 2:00 AM
Subject: [opendemo-devel] file format question


> Hello!
>
> What happens, if the binary part of an entity contains a '\0' or even
stranger
> a '<' character? Is this possible at all?
>
> Bye, Uwe
>
>
> _______________________________________________
> opendemo-devel mailing list
> ope...@li...
> https://lists.sourceforge.net/lists/listinfo/opendemo-devel
>

[opendemo-devel] make system

[Dr. U. G. <Uwe...@ph...>]

Hello!

Shouldn't we try to reorganize the make system? I hate it to have a Makefile
for Linux native, a shell-script for bytecode compiled under Linux, a 
batch file for bytecode compiled under Windows and a project file for native
Windows code. Any change of source file names must be reflected in in all these
places! Woudn't it be much better to have a standard Makefile with different 
sets for compiler and compiler flags.
And BTW, changes in header files don't trigger recompiles of any source files,
where these header files are included in the current system.

Bye, Uwe

[opendemo-devel] file format question

[Dr. U. G. <Uwe...@ph...>]

Hello!

What happens, if the binary part of an entity contains a '\0' or even stranger
a '<' character? Is this possible at all?

Bye, Uwe

[opendemo-devel] I'm back

[Conor D. <co...@ma...>]

Well it's nice to know that interest has picked up again in opendemo! My
planetquake address doesn't like me, so I didn't notice any activity on the
list until the blurb on machinima.com (no excuses for the week delay,
sorry).

I'll try to start working again later this week if possible.

-Conor

[opendemo-devel] finally checked it out: it works!

[Dr. U. G. <Uwe...@ph...>]

Hello!

I finally found the time (it's heavily raining outside) to check out the
current state of OpenDemo: It works with Linux with the native and the
byte code libraries.

The overall handling (od_play, od_demofile setting, map changing etc.) is a
bit clumsy and I'm thinking about a method to simplify it.

Really good work, Conor!

Bye, Uwe

[opendemo-devel] compile warnings mostly removed

[Dr. U. G. <Uwe...@ph...>]

Hello!

I just removed most of the compiler warnings. Some really strange
stuff like
../odr_main.c:387: warning: expression with no effect elided                                (this is the usage of the OD_SETBIT macro)
and void* -> function* casts remain but it is a start.
Some functions, which should return always a value have pure 'return;' statements
or they simply end without return. I have no idea, what I should correct there.

I get now some error messages, that I did not had yesterday. Is this q3lcc somehow
non-deterministic?

Now I can finally try it out.

Bye, Uwe

[opendemo-devel] byte code generation under Linux

[Dr. U. G. <Uwe...@ph...>]

Hello!

I just checked in new shell scripts to generate the bytecode
under Linux. I also added the tool set from Loki for Linux
to make it possible. The compiling gives some warnings, I'll
remove them soon.

Bye, Uwe

Re: [opendemo-devel] linux makefiles

[Dr. U. G. <Uwe...@ph...>]

On Mon, Apr 16, 2001 at 11:08:56PM -0500, Conor Davis wrote:
> A few weeks back i committed a fairly functional version of opendemo to the
> cvs, but without any linux makefiles (only MSVC project files). I had
> intended to work on the makefiles, but I got frustrated and gave up. So if
> anyone can hack together a few makefiles, it would be great.
I just added opendemo/src/q3a/opendemo/Makefile to CVS. It can compile
native shared libraries on a usual Linux system. I did not yet try to
load them actually in Q3A. I removed some warnings on the way. GCC is 
(even without any -Wall switches) sometimes very picky about 'allowed'
C code.

The base for my Makefile was the Makefile out of the Linux-SDK in the
file linuxq3sdk-1.1b.x86.run somewhere on Liki's ftp server.

The next step is to bring the bytecode generators to work under Linux
and then I'll finally try it out.

Uwe

[opendemo-devel] linux makefiles

[Conor D. <ce...@pl...>]

A few weeks back i committed a fairly functional version of opendemo to the
cvs, but without any linux makefiles (only MSVC project files). I had
intended to work on the makefiles, but I got frustrated and gave up. So if
anyone can hack together a few makefiles, it would be great.

-Conor

[opendemo-devel] OpenDemo happenings

[Conor D. <ce...@pl...>]

Hello,

It's been a while since there was activity on this list, but I just wanted
to say I haven't been dead the past few months. I've rewritten the odrecord
module so that it writes complete (plain-text, not wbxml) XML files (with
entity and player info and such). Also, I wrote the odplay module to read
the demo file and play it (sort of). Before I commit it to CVS, I feel it is
important to finalize the directory structure. In my working directory, I
have it set up like this:

opendemo/
    bin/
        q3a/
            cpp.exe
            lcc.exe
            q3asm.exe
            rcc.exe
    doc/
        <documentation files>
    src/
        q3a/
            common/
                od_common.c
                od_common.h
                od_filebuf.c
                od_filebuf.h
                od_mem.c
                od_mem.h
                od_membuf.c
                od_membuf.h
                od_xml_read.c
                od_xml_read.h
            include/
                <q3a include files>
            odplay/
                cgame/
                    <odplay cgame files>
                game/
                    <odplay game files>
                    odp_main.c
                    odp_parse.c
                    odp_private.h
                    odp_shared.h
                ui/
                    <odplay ui files>
            odrecord/
                game/
                    <odrecord game files>
                    odr_main.c
                    odr_private.h
                    odr_shared.h

Eventually I think it will be necessary to combine the odrecord and odplay
modules into one that functions differently depending on how a few cvars are
set. This is because:
a) right now I have to manually copy demo files from the odrecord directory
to the odplay directory, something which most people probably do not want to
do
b) while it is techincally possible to use one generic odplay game module
for any type of demo, game-specific odplay modules would be better

This would make the directory tree like:
opendemo/
    ...
    src/
        q3a/
            common/
            opendemo/
                game/
                cgame/
                ui/

Also, right now it is based off the 1.17 source, and I think it would be
best to "do it right" by initializing the CVS tree with the 1.27 source
(with an appropriate "id" branch, "id_1_27" tag), then commit the opendemo
source.

Regards,
Conor Davis
ce...@pl...

[opendemo-devel] SAX vs. proprietary XML api

[Conor D. <co...@ma...>]

I got to thinking about the demo format for OpenDemo. I can see two
different ways to go about reading/writing an odxml document:

a) Read/write to plain-text XML using the SAX (for reading at least)
interface. The wbxml format could be used to shrink the XML file a bit. This
approach has the benefit of using standard, established API's, so utility
writers might have an easier time to develop their programs.

b) Use a proprietary API to write text/binary xml files. This would expand
on the wbxml concept. In addition to storing tags in binary format, the
actual
content (such as configstring indicies and vectors) could be stored in
binary, so the demo file is even smaller. Also, if we change the API a bit
we could greatly improve the speed of parsing the file. Instead of doing a
strcmp() for each tag and attribute name, we could compare the tag's index
(supplied when opening the xml file) to the index of the tag we are looking
for. And because I can code better than I can talk, here is an example:

With SAX:
void
startElement(void *userdata, const CHAR *name, const CHAR **attrs)
{
    ...
    if (!strcmp(name, "mytag"))
    {
        for (i = 0; attrs[i] != NULL; i++)
        {
            const CHAR *aname = attrs[i++];
            const CHAR *avalue = attrs[i];

            if (!strcmp(aname, "my_integer_attr"))
            {
                int myint = strol(avalue);
            }
            else if (!strcmp(aname, "my_string_attr))
            {
                const CHAR *mystring = avalue;
            }
            else
            {
                printf("Unknown tag name: %s\n", name);
            }
        }
    }
    ...
}

Proprietary API:
typedef struct
{
    int id;
    const char *string;
} odxml_tag_t;

/*
    Helper functions GetInteget, GetFloat, and GetString can
    convert from the type in which it was stored to the type that
    was requested. For example, calling GetInteger on an
    odxml_data_t object with stored_type==TYPE_STRING
    would convert the string data to an int with strtol().
*/
typedef struct
{
    enum
    {
        TYPE_INT,           /* stored in binary format in xml file */
        TYPE_FLOAT,    /* ditto */
        TYPE_STRING   /* stored in text format in xml file */
    } stored_type;
    union
    {
        int  integer;
        float float;
        char *string;
    } data;
} odxml_data_t;

void
startElement(void *userdata, const odxml_tag_t *name, const odxml_tag_t
*attr_names, const odxml_data_t *attr_values)
{
    ...
    if (name->id == MYTAG_INDEX)
    {
        for (i = 0; attr_names[i].string != NULL; i++)
        {
            const odxml_tag_t *aname = attr_names + i;
            const odxml_data_t *avalue = attr_values + i;

            switch (aname->id)
            {
            case MY_INTEGER_INDEX:
                int myint = GetInteger(avalue);
                break;
            case MY_STRING_INDEX:
                const CHAR *mystring = GetString(avalue);
                break;
            default:
                /* we can compare with strcmp() here if we wanted to */
                printf("Unknown tag name: %s\n", aname->string);
            }
        }
    }
}

This piece of code could work for binary XML files and plain-text XML files
(but not as fast). Since most of the data we are storing is numbers anyway,
this would help to avoid many strtol() and itoa() calls (if the file is in
binary format) in addition to reducing the file size. My big concern here is
programs such as Keygrip already take awhile to load large dm2 files (which
are compact), and having a text-only format might make load times
unbearable.

Conor Davis
ce...@pl...

Re: [opendemo-devel] new Makefile (was: misc. ramblings)

[Conor D. <ce...@pl...>]

> > First, about the cvs layout: I propose to change the directory hierarchy
of
> > the mods/ directory to this:
> > [...]
> Sounds good for future game support. Who should do it?

Not me!

> > P.S. Uwe: is there any way you can remove the /mods/ top-level directory
> > from cvs? I goofed when trying to create a branch for the original 1.17
> > source and put it in in /mods/ instead of /opendemo/mods/ ; I was able
to
> > remove the files but not the directories.
> You can't remove directories from CVS. You can remove the files and use
> cvs update -P
> (for prune). This will remove empty directories. BTW, the vm directories
> in game, cgame and ui will be removed is this way too. It may be better
> to checkin an empty file '.keepme' in these directories.

I figured admins might have shell access to their cvs repository. Anyway,
the /mods/ directory does show up in the web interface and can confuse
people (and make me look bad!).

I recently committed a rewrite of the odrecord Makefile which lets you
compile in qvm or native mode. You first have to do a 'make configure' to
write your MODE and TARGET variables to config.mak so you don't need to put
them in the command-line every time. Using this system, I was able to
compile in both qvm and native modes, but qagame.qvm wouldn't load under
quake3.

The makefile is rather large and should probably be replaced by a system of
common sub-makefiles (qvm.mak, debug.mak, etc.), but I think that should
wait until after the directory re-structuring is finished.

Conor Davis
ce...@pl...

Re: [opendemo-devel] misc. ramblings

[Uwe G. <Uwe...@ph...>]

Hello!

> First, about the cvs layout: I propose to change the directory hierarchy of
> the mods/ directory to this:
> opendemo/
>     mods/
>         q3a/
>             bin/
>             include/
>             odrecord/
>                 cgame/
>                 game/
>                 ui/
>             odplay/
>                 cgame/
>                 game/
>                 ui/
Sounds good for future game support. Who should do it?

> (Uwe, is there any reason for the source/ directories?).
It was there in the original source package from id.

> I'd also like to separate the web/ directory into a different package. It
> seems pretty unrelated to the rest of the opendemo source.
You may be right. As I'm the only one, who can upload the web-page to
machinima anyway, we can leave it out. I won't create another SourceForge
project. I simply let it in my private CVS tree at home.

> Finally, I've been trying to re-do the odrecord Makefile to support both
> qvm's and native libraries. This should speed up compilation under *nix
> because Wine/lcc.exe takes forever to compile on my pc.
The wine startup-time is much too high. I'm thinking about a stripped down
version of wine but in the end, the best way is a native compiler.
In the current beta of the "Quake III Arena SDK" from Loki was no compiler
but the hint in the README:
# Binary versions of the q3asm and lcc tools
# required to build QVM bytecode, as well as
# current versions of the game/cgame sources
# will be added to the SDK in the next release.                                   
> I figure a debug
> version of the native libraries would be the most useful,
I did not try to debug mods with a debugger but it could be fun.

> but do you see any
> need for optimize or profile versions?
If we are too slow, profiling would be nice.

> P.S. Uwe: is there any way you can remove the /mods/ top-level directory
> from cvs? I goofed when trying to create a branch for the original 1.17
> source and put it in in /mods/ instead of /opendemo/mods/ ; I was able to
> remove the files but not the directories.
You can't remove directories from CVS. You can remove the files and use
cvs update -P 
(for prune). This will remove empty directories. BTW, the vm directories 
in game, cgame and ui will be removed is this way too. It may be better
to checkin an empty file '.keepme' in these directories.

Another point, please send emails to the mailing list from the member address
and not from the outside. I have to approve all outside mails. You can also
resubscribe from your current address.

Uwe

[opendemo-devel] misc. ramblings

[Conor D. <ce...@ep...>]

Hello:

First, about the cvs layout: I propose to change the directory hierarchy of
the mods/ directory to this:
opendemo/
    mods/
        q3a/
            bin/
            include/
            odrecord/
                cgame/
                game/
                ui/
            odplay/
                cgame/
                game/
                ui/

This means adding a q3a/ directory and removing the source/ directories
(Uwe, is there any reason for the source/ directories?).

I'd also like to separate the web/ directory into a different package. It
seems pretty unrelated to the rest of the opendemo source.

Finally, I've been trying to re-do the odrecord Makefile to support both
qvm's and native libraries. This should speed up compilation under *nix
because Wine/lcc.exe takes forever to compile on my pc. I figure a debug
version of the native libraries would be the most useful, but do you see any
need for optimize or profile versions?

Conor Davis
ce...@ep...

P.S. Uwe: is there any way you can remove the /mods/ top-level directory
from cvs? I goofed when trying to create a branch for the original 1.17
source and put it in in /mods/ instead of /opendemo/mods/ ; I was able to
remove the files but not the directories.

[opendemo-devel] 1.17 code from id tagged with id_1_17

[Uwe G. <Uwe...@ph...>]

Just to start the list:

The revisions of all source files in mods/odrecord and mods/odplay which came
from id with the source release 1.17 are now tagged with the symbol "id_1_17".
So it is easier to make a global patch relative to a globally known code-base.

Uwe