opendemo-devel Mailing List for OpenDemo (Page 3)
Status: Beta
Brought to you by:
girlich
Menu
▾
▴
opendemo-devel
You can subscribe to this list here.
| 2000 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(4) |
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
(1) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
(3) |
Sep
(12) |
Oct
(7) |
Nov
(2) |
Dec
(10) |
| 2002 |
Jan
(52) |
Feb
|
Mar
(1) |
Apr
(6) |
May
(1) |
Jun
(5) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2003 |
Jan
(5) |
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
(1) |
| 2004 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Conor D. <co...@ma...> - 2002-01-17 06:43:27
|
Hello, ----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> To: "OpenDemo Development List" <ope...@li...> Sent: Sunday, January 13, 2002 1:55 PM Subject: [opendemo-devel] qvm playback with compressed files > Hello! > > I just analysed the crashing behaviour of the QVM code on playback of > compressed files. It is really strange. The attached patch file adds a mini > trace to the first call of fbReadChar, which get the character '<', as we all > would expect. But the calling function xmlParseText get another return value. Ok this is driving me crazy. If vm_game is 1, then I don't get the bug you described, but I do get the ~ operator bug. If vm_game is 2, then I do get the bug you described (with the return value of gzread, it crashes before fbReadChar exits), but I don't get the ~ operator bug. I fixed the ~ bug, but nothing I do fixes the return value bug. Perhaps the virtual machine's frame pointer is getting corrupted somehow? I have no idea how to test for this. BTW, gzip input does work for me with vm_game=1, at about half the performance of a native dll. If we convert a few more functions to macros (fbEOF and gzeof come to mind), maybe it will get better. -Conor |
|
From: Conor D. <co...@ma...> - 2002-01-17 01:28:46
|
Hello, I saw this site today (http://www.gamerstv.net/news.php) and it looks like something I've wanted to do for a long time. It is a system that allows live spectating of quake3 matches, where clients connect to one of many spectator-servers, which are in turn linked to the server that actually hosts the match. This was a goal I had for Quake2 Relay (hence the name 'Relay'), but I never got around to implementing it. It's also why I used q2-style delta-encoding (baselines, sequence numbers, 16 snapshot states), in case we wanted to do something like this in the future. Do you guys think this is a long-term goal worth pursuing? -Conor |
|
From: Eugene 'H. B. <hm...@ma...> - 2002-01-16 07:10:04
|
-----Original Message----- > (hopefully) working version. Please try it out and do some performance > tests. I have to admit, that it did not really feel any faster. I am now in another city and far from my machine (do not have notebook yet), so I can't do any profiling at all. I'll be back no earlier than in one week. Sadly, you will have to find a way to set up profiling on linux... Anyway, investing time in programming ring buffer is a great thing to do by you, Uwe. Can't test it on my machine yet... ------ - Eugene 'HMage' Bujak |
|
From: Eugene 'H. B. <hm...@ma...> - 2002-01-15 15:08:05
|
> > > rtcw can't clientrecord - this will be our chance) > > That's an interesting point. I always thought about so different games beside > > Q3A, that all sources would be useless anyway but RTCW is almost Q3A. Now > > I'm really confused... > I thought a bit about all this and think we can keep zlib and also the other > stuff in the q3a directory. Where some file is located, is totally irrelevant. But it might be confusing for people trying to integrate opendemo in their mods, and, as far as I understand, having rtcw mod source in separate directory will not be a problem with 'two copies of the same file' as they are not the same and come from two 'different' places. > q_math.c etc). If we add rtcw, we simply get (only in Makefile rules, no copy > of C sources are allowed) the relevant files from the q3a directory and we are > ready. I really don't think that we can use Q3 mod sources in rtcw mod sources, even if they are almost the same, they are to be kept separatedly... Or I don't understand you? > > It feeled much more jerky, than before. > I'm not really sure anymore about this: during all my replay checks I compiled > in the background the Transgaming WineX source code. Without this additional > load, it feeled much better. I think I need some more objectivity (FPS). Enabling cg_drawfps 1 and setting com_maxfps to one thousand will do the trick... This will bring some sort of objectivity. Can't profile yet because I'm far from home. (sitting in internet cafe now) ------ - Eugene 'HMage' Bujak |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-15 07:17:46
|
Hello!
> I think we should consider changing the file format as follows
> to speed up playback,
Sounds good.
> use much less memory,
Do we have a problem here? Has the QVM a memory restriction?
> and simplify things:
I'm always for the most simplified solution.
> - Record only one map per xml file, with each xml possilbly including a
> reference to the next demo.
Was it ever possible to record more than one map? It is possible to play more
than one map in Q1 and QW and Q2 but all important movies I'm aware of, use
this reference technique to simplify demo editing. Even games like Half-Life,
where one level is really small have to reload the next level. If the demo file
changes as this time too is not so important.
> We really need to do this if we want to allow
> rewinding of demos during playback, since we can't store demos in memory. I
> don't see how this hurts functionality.
So you think about using odseek() to get to a special moment in the past?
> - Get rid of the 16 states used for delta encoding, and force delta'ing only
> from the previous snapshot (or baselines). This reduces the size of the od
> struct by about 9 megs, and should completely remove the odpStartElement
> performance bottleneck. We can keep the "currentSeq" and "deltaSeq" xml
> attributes, just in case.
That is reasonable. We can't miss a single frame, we are the server after all,
so we can delta encode as we like it best. I never understood which one of
delta encoding against the previous snapshot or baselines is any better. How
can the recording mechanism decide this?
> - Maybe move element data into attributes? E.g instead of:
> <currentOrigin index="0">586.87023</currentOrigin>
> we could use:
> <currentOrigin index="0" value="586.87023" />
> I say this because AFAIK there is less variation allowed for attributes than
> character data. For example, I think leading and trailing whitespace must
> always be removed from attributes,
That's true only for non-CDATA attributes.
As short look into our DTD reveals, we use enumeration type (0|1|2) or NMTOKEN.
With the production rules from XML (here S denotes white space)
[3] S ::= (#x20 | #x9 | #xD | #xA)+
[4] NameChar ::= Letter | Digit | '.' | '-' | '_' | ':' |
CombiningChar | Extender
[5] Name ::= (Letter | '_' | ':') (NameChar)*
[6] Names ::= Name (S Name)*
[7] Nmtoken ::= (NameChar)+
[8] Nmtokens ::= Nmtoken (S Nmtoken)*
we see clearly, that you are right. There are no white-spaces at both ends.
So I would even think, leading and trailing attribute white spaces let the XML
file in the state "well-formed" but it is no longer conformant with our DTD.
Just for completeness, I choose NMTOKEN because it has some kind of restriction
over CDATA but there is no "floating point" attribute type, which would have
been better.
> while character data may or may not need
> that whitespace (depending on the DTD).
And the attribue xml:space.
> Since we don't use a validating XML
> parser to handle these variances, maybe we should move to attributes?
If it simplifies something, go for it.
I'll write a XSL stylesheet to convert old files into new ones (if we finally
decide on the new format). That is much better than supporting both formats or
writing a dedicated application for this task. There are so many free XSLT
processors available (http://www.xmlsoftware.com/xslt/), it should be no
problem to get one of them running on every computer system.
But we should never forget, that we hope to reuse much of all this framework
for other games as well. Maybe the names of the entity parameters are called
a bit different but all this XML file format stuff and delta-encoding technique
must be reusable at least for RTCW and maybe also also for DOOM3.
Bye, Uwe
|
|
From: Conor D. <co...@ma...> - 2002-01-15 07:15:24
|
I think I've found abother bug in the qvm compiler. If you use ~var, where var is a variable (or an expression that contains a variable) and not a compile-time constant, you get funky values returned. I'm not sure if this could cause the results you got, but zlib does use ~ with variables which could cause problems. I also modified the filebuf to read more data at once. Before, it would call gzread for only a few characters at a time (usually just 1), because the membuffer never grew to more than a dozen bytes if even that. Since each gzread triggers about 5 inflate functions each call, this was really bogging things down. There's also an extensive function tracing system that logs most of the important function calls (while slowing everything down to 2 fps). Helpful for tracking qvm crashes (and how I found the filebuf problem). I'll commit all the fixes tomorrow; I'm too tired right now. -Conor ----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> To: "OpenDemo Development List" <ope...@li...> Sent: Sunday, January 13, 2002 1:55 PM Subject: [opendemo-devel] qvm playback with compressed files > Hello! > > I just analysed the crashing behaviour of the QVM code on playback of > compressed files. It is really strange. The attached patch file adds a mini > trace to the first call of fbReadChar, which get the character '<', as we all > would expect. But the calling function xmlParseText get another return value. > > Loading vm file vm/qagame.qvm. > VM file qagame compiled to 1493476 bytes of code > qagame loaded in 19045056 bytes on the hunk > fbReadChar(0) = '<' = 60 > xmlParseText: character = 'h' = 104 > End of recording > Received signal 11, exiting... > Shutdown tty console > > Sooner or or later this results in XML problems and the playback stops > (output: "End of recording"). Then somehow (not really important just now) > the program crashs totally with a SIGSEGV (segmantation violation). > > So the main problem is why the print of the character before the return > statement of fbReadChar() was '<' but the calling function get another value > back. I have no idea, how this could occur, especially I don't see the > connection with the zlib code. > > Bye, Uwe > |
|
From: Conor D. <co...@ma...> - 2002-01-14 17:05:10
|
Hello all,
I think we should consider changing the file format as follows to speed up
playback, use much less memory, and simplify things:
- Record only one map per xml file, with each xml possilbly including a
reference to the next demo. We really need to do this if we want to allow
rewinding of demos during playback, since we can't store demos in memory. I
don't see how this hurts functionality.
- Get rid of the 16 states used for delta encoding, and force delta'ing only
from the previous snapshot (or baselines). This reduces the size of the od
struct by about 9 megs, and should completely remove the odpStartElement
performance bottleneck. We can keep the "currentSeq" and "deltaSeq" xml
attributes, just in case.
- Maybe move element data into attributes? E.g instead of:
<currentOrigin index="0">586.87023</currentOrigin>
we could use:
<currentOrigin index="0" value="586.87023" />
I say this because AFAIK there is less variation allowed for attributes than
character data. For example, I think leading and trailing whitespace must
always be removed from attributes, while character data may or may not need
that whitespace (depending on the DTD). Since we don't use a validating XML
parser to handle these variances, maybe we should move to attributes?
-Conor
|
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-13 19:55:25
|
Hello! I just analysed the crashing behaviour of the QVM code on playback of compressed files. It is really strange. The attached patch file adds a mini trace to the first call of fbReadChar, which get the character '<', as we all would expect. But the calling function xmlParseText get another return value. Loading vm file vm/qagame.qvm. VM file qagame compiled to 1493476 bytes of code qagame loaded in 19045056 bytes on the hunk fbReadChar(0) = '<' = 60 xmlParseText: character = 'h' = 104 End of recording Received signal 11, exiting... Shutdown tty console Sooner or or later this results in XML problems and the playback stops (output: "End of recording"). Then somehow (not really important just now) the program crashs totally with a SIGSEGV (segmantation violation). So the main problem is why the print of the character before the return statement of fbReadChar() was '<' but the calling function get another value back. I have no idea, how this could occur, especially I don't see the connection with the zlib code. Bye, Uwe |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-13 14:39:22
|
Hello! On Sun, Jan 13, 2002 at 01:27:07PM +0100, Dr. Uwe Girlich wrote: > On Sun, Jan 13, 2002 at 01:57:58PM +0300, Eugene 'HMage' Bujak wrote: > > There's one thing to remember - we do not want to be quake3-only project, as > > soon as rtcw mod sources come out, I plan to test it there (as far as I know, > > rtcw can't clientrecord - this will be our chance) > That's an interesting point. I always thought about so different games beside > Q3A, that all sources would be useless anyway but RTCW is almost Q3A. Now > I'm really confused... I thought a bit about all this and think we can keep zlib and also the other stuff in the q3a directory. Where some file is located, is totally irrelevant. My 'integrated development environment' (cscope+vi) will find it anyway and I hope this is true for more sophisticated tools. If some rules in the Makefile contain some more paths or not is also not so important. Even now we have several files, whose are compiled in every subdirectory but they are located only in the game directory (bg_misc.c, q_math.c etc). If we add rtcw, we simply get (only in Makefile rules, no copy of C sources are allowed) the relevant files from the q3a directory and we are ready. > QVM code seems a bit sluggish now, that we have > fbReadChar -> fbGrow -> gzread -> fread -> odfread -> trap_FS_Read > (OK fread and odfread are one (macro)). It feeled much more jerky, than > before. I'm not really sure anymore about this: during all my replay checks I compiled in the background the Transgaming WineX source code. Without this additional load, it feeled much better. I think I need some more objectivity (FPS). Bye, Uwe |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-13 12:26:49
|
Hello! On Sun, Jan 13, 2002 at 01:57:58PM +0300, Eugene 'HMage' Bujak wrote: > There's one thing to remember - we do not want to be quake3-only project, as > soon as rtcw mod sources come out, I plan to test it there (as far as I know, > rtcw can't clientrecord - this will be our chance) That's an interesting point. I always thought about so different games beside Q3A, that all sources would be useless anyway but RTCW is almost Q3A. Now I'm really confused... > Sorry can't check your changes yet because I'm far from home. But I did. I corrected a bit in the Makefile to create working game.(bat|q3asm) and game_ta.(bat|q3asm) files. Native code can replay plain text and gzip compressed files. QVM code can replay plain text and crashes Q3A with sig 11 on gzip compressed files. I'll think about it. QVM code seems a bit sluggish now, that we have fbReadChar -> fbGrow -> gzread -> fread -> odfread -> trap_FS_Read (OK fread and odfread are one (macro)). It feeled much more jerky, than before. Would it help, if we make the gzip stuff optional (at compile time, with macros)? Bye, Uwe |
|
From: Eugene 'H. B. <hm...@ma...> - 2002-01-13 10:59:27
|
> We could equally put everything in the main game directory. There's one thing to remember - we do not want to be quake3-only project, as soon as rtcw mod sources come out, I plan to test it there (as far as I know, rtcw can't clientrecord - this will be our chance) > I think you just convinced me, that opendemo/src/q3a/opendemo/game/zlib is a > better place? Bad thing to do(tm). If we will be only-quake3, we will have some legal problems coming. Our agenda should be the same - the opendemo is a open format, and quake3, incidently, is the only supported game platform *yet*. Adding this to RTCW and (hopefully :) Doom3 should prove our agenda. Anyway, we need abstraction. I don't think that there is that much perfomance hit on that, and we have some advantage... Sorry can't check your changes yet because I'm far from home. ------ - Eugene 'HMage' Bujak |
|
From: Conor D. <co...@ma...> - 2002-01-12 08:05:30
|
Greetings, I added the zlib stuff to /src/q3a/opendemo/game/zlib and adjusted the project files and Makefiles accordingly. I also moved everything in /src/libs to /src/q3a/opendemo/game. I also added a q3asm wrapper shell script as /bin/q3a/q3asm.sh, because using q3asm directly from the Makefile (the win32 version at least) would not work. It would try to write to -o.qvm instead of qagame.qvm. I didn't change the makefile to use it, because maybe it is just my setup. I commited the changes in a rush, so there may be more bugs than before. It compiles at least. I've heard that the qvm chokes at run-time on fall-through switch/case statements (i.e. don't end with a break statement), but I haven't changed the zlib files to fix that. I noticed that the in-source copyrights were GPL, so I changed them to LGPL. I also added Uwe and Eugene to the copyrights on files that I knew they had changed (double-check me on them). -Conor |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-11 21:53:36
|
Hello! I finally found the small bug in my ring-buffer code and checked in a (hopefully) working version. Please try it out and do some performance tests. I have to admit, that it did not really feel any faster. opendemo/src/libs/od_filebuf.h 1.3 -> 1.4 opendemo/src/libs/od_filebuf.c 1.3 -> 1.4 fbShrink() is now useless, as fbSkip() alone advances the read pointer (tail) and makes the ring buffer smaller. I did not implement any resizing to conserve memory: if an fbGrow() request is too big, the ring buffer grows accordingly with mbResize() but if we have nothing more to read from the buffer, it won't reduce its memory buffer size. All this odRealloc() stuff was really too much. Bye, Uwe |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-11 13:31:00
|
Hello! > As I said in my earlier message, we could just move everything inside the > mod's game directory and use them just for quake3 mods. Is the stuff in > there really that useful outside of quake3? You're totally right. A memory buffer, file-buffer or a XML parser aren't so special. There are other full-featured software projects out there dealing only with this kind of stuff. We could equally put everything in the main game directory. > > > The zlib stuff shares some functionality with > > > od_filebuf.c, so maybe the reimplimentation could combine the two. > > I think odFread() should be replaces by gzread() and nothing more. This > > can easily be done before or after the ring-buffer comes in place. > I had thought that zlib kept its own buffer for decompressed data, but it > appears that it does not. Even if it keeps its own buffer, the API is gzread() only. I'm sure zlib has a backlog buffer; thats the base idea of decompression: pointer to already found material some bytes ago. If you increase this window, you get better compression but the zlib standard API is gzread() as a direct replacement for read(). > I'll commit my code if you commit yours... :) I have one hour this evening and maybe two tomorrow. This should be enough. > I'll move the zlib code over from the test mod to the opendemo tree (in > opendemo/src/libs) tomorrow. I think you just convinced me, that opendemo/src/q3a/opendemo/game/zlib is a better place? But it is your code and your directory layout anyway. Bye, Uwe |
|
From: Conor D. <co...@ma...> - 2002-01-11 08:40:27
|
Hello, ----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> To: "OpenDemo Development List" <ope...@li...> Sent: Friday, January 11, 2002 1:06 AM Subject: Re: [opendemo-devel] zlib > [...] > > If you guys want to hack at it, I can commit it to cvs even > > though it is broken (all the zlib stuff is in the opendemo/game/zlib > > directory). > I did not find a zlib directory in opendem/src/q3a/opendemo/game/ and we don't > have a opendemo/game directory at all. So did you check it in already? > I would prefer opendem/src/libs/zlib as a more generic place. I meant that was the layout in my working directory (full path is opendemo/src/q3a/opendemo/game/zlib). IMO opendemo/src/libs/ should be for code that is not quake3-specific. The stuff that is in there right now can be used in other things, but right now it is hard for them to tell if they should #include <stdio.h> or #include "q_shared.h" (#ifdef Q3_VM is not good enough because "q_shared.h" should be used when compiling a native library for a quake3 mod). As I said in my earlier message, we could just move everything inside the mod's game directory and use them just for quake3 mods. Is the stuff in there really that useful outside of quake3? > > Everything else should still work if you don't try to use a gzip > > file. > So you used the standard zlib interface, which reverts to the plain file > behaviour, if the file was not compressed. Correct. > [...] > > The zlib stuff shares some functionality with > > od_filebuf.c, so maybe the reimplimentation could combine the two. > I think odFread() should be replaces by gzread() and nothing more. This can > easily be done before or after the ring-buffer comes in place. I had thought that zlib kept its own buffer for decompressed data, but it appears that it does not. > My ring-buffer comes along nicely, the code is completely typed in, compiles > without warnings with GCC and qvm and does not crash, but it doesn't do > anything useful yet. I hope for a bit more debugging time over the weekend. I'll commit my code if you commit yours... :) I'll move the zlib code over from the test mod to the opendemo tree (in opendemo/src/libs) tomorrow. -Conor |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-11 07:05:49
|
Hello! > I've tried to add zlib support to a test mod, Yes! Nobody can say any longer: your XML text files are too long! > ... but it refuses to work when > compiled in qvm mode. We can find the problem, I'm sure. > It either bombs out to the console or crashes quake3 > altogether. Sounds interesting. > If you guys want to hack at it, I can commit it to cvs even > though it is broken (all the zlib stuff is in the opendemo/game/zlib > directory). I did not find a zlib directory in opendem/src/q3a/opendemo/game/ and we don't have a opendemo/game directory at all. So did you check it in already? I would prefer opendem/src/libs/zlib as a more generic place. > Everything else should still work if you don't try to use a gzip > file. So you used the standard zlib interface, which reverts to the plain file behaviour, if the file was not compressed. > I'm looking into just making my own deflate/inflate routines (it's something > I've wanted to do anyway). You're brave! > The zlib stuff shares some functionality with > od_filebuf.c, so maybe the reimplimentation could combine the two. I think odFread() should be replaces by gzread() and nothing more. This can easily be done before or after the ring-buffer comes in place. My ring-buffer comes along nicely, the code is completely typed in, compiles without warnings with GCC and qvm and does not crash, but it doesn't do anything useful yet. I hope for a bit more debugging time over the weekend. Bye, Uwe |
|
From: Conor D. <co...@ma...> - 2002-01-11 03:55:57
|
Hello, I've tried to add zlib support to a test mod, but it refuses to work when compiled in qvm mode. It either bombs out to the console or crashes quake3 altogether. If you guys want to hack at it, I can commit it to cvs even though it is broken (all the zlib stuff is in the opendemo/game/zlib directory). Everything else should still work if you don't try to use a gzip file. I'm looking into just making my own deflate/inflate routines (it's something I've wanted to do anyway). The zlib stuff shares some functionality with od_filebuf.c, so maybe the reimplimentation could combine the two. -Conor |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-09 21:05:01
|
Hello! > ... I intend to > implement some ring-buffer code tomorrow and 'll fix this one too. xml parsing is cleaned up (uses only allowed fb-interface) in CVS. The ring buffer is almost ready; it would take about one more hour but my concentration is too low to continue now. Bye, Uwe |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-08 18:44:45
|
Hello!
> I come to the conclusion, that I first have to look at and understand all the
> typical uses of all the fb* functions.
I found a possible buffer overrun in the code:
libs/od_xml_read.c/xmlParseCharacters()
reads characters up to the first processing character (<,&) and
discards any whitespaces up to the first real character.
The main point is
firstChar = -1;
for (len = 0; ; len++) {
c = fbReadChar(doc.fb, len);
if (c == '<' || c == '&')
break;
...
if (!IS_BLANK(c) && firstChar == -1)
firstChar = len;
}
So we have now len bytes checked and at the position firstChar
(firstChar <= len) ends the whitespaces.
If we had only 'len' whitespaces, these 'len' bytes will be ignored:
if (firstChar == -1) {
if (doc.sax && doc.sax->ignorableWhitespace)
doc.sax->ignorableWhitespace(doc.userData,
mbGetBuffer(doc.fb->mb) + doc.fb->offset, len);
}
else {
But if we had also real data, we do something strange:
if (doc.sax && doc.sax->characters)
doc.sax->characters(doc.userData,
mbGetBuffer(doc.fb->mb) + doc.fb->offset + firstChar, len);
}
This would mean, we have len characters after the firstChar position.
I think, this must be:
else {
/*
ignorable whitespace comes only between 2 tags,
so we have no doc.sax->ignorableWhitespace() call here.
*/
/* after the white-spaces come only len-firstChar characters */
if (doc.sax && doc.sax->characters)
doc.sax->characters(doc.userData,
mbGetBuffer(doc.fb->mb) + doc.fb->offset + firstChar, len - firstChar);
}
This problem never arose, because the ODQ3A files contain no simple characters
but only XML tags and everything is in attributes.
I'll read the code a bit further, maybe I find something more. I intend to
implement some ring-buffer code tomorrow and 'll fix this one too.
Bye, Uwe
|
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-08 10:05:09
|
Hello! A simple ring buffer (with a simple ring_read() and ring_write() function and nothing more) is not good enough, because our read functions don't advance the tail pointer. We do a lot of string parsing (with an open end) with the data in the buffer, so we need the data in the ring buffer even after a read already occured. Only the fbSkip() function discards the old data. So we possibly need more and more space in the ring buffer, which makes its organization not exactly easy. I come to the conclusion, that I first have to look at and understand all the typical uses of all the fb* functions. Maybe I can rewrite some of them to make a simpler API than the current fb* possible. Bye, Uwe |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-07 16:35:28
|
Hello!
> > A ring-buffer would simply solve this bottleneck of constantly calling
> > fbGrow(), mbResize and memcpy() but we have 21 occurrences of
> > mbGetBuffer(),
> > whose are not so easily converted to a ring-buffer. We need to know before
> > calling mbGetBuffer(), how many bytes should be available. Maybe this is
> > already done with the usually fbGrow() call before mbGetBuffer() but I'm
> > not really sure in all 21 cases.
> If you put the ring-buffer code in od_filebuf.c instead of od_membuf.c (or
> add a middle layer), then that shouldn't be a problem. The fb* interface
> does not provide direct access to the buffer, so changing from linear to
> ring buffers shouldn't require other changes. mbGetBuffer() just gives you
> the start of the memory block (same as any other ring-buffer).
Almost correct. I thought mb* functions are used by fb* functions only but
that's not the case. Many others use mb* stuff too (and with it mbGetBuffer().
So we have to concentrate on fb* functions only.
But what is with xmlParseCharacters(), where we have the code:
if (firstChar == -1) {
if (doc.sax && doc.sax->ignorableWhitespace)
doc.sax->ignorableWhitespace(doc.userData,
mbGetBuffer(doc.fb->mb) + doc.fb->offset, len);
}
else {
if (doc.sax && doc.sax->characters)
doc.sax->characters(doc.userData,
mbGetBuffer(doc.fb->mb) + doc.fb->offset + firstChar, len);
}
This dissolves all the layering and gives a sax callback function direct access
to the (hopefully long enough) memory buffer.
I need to think about all this a bit more.
Bye, Uwe
|
|
From: Conor D. <co...@ma...> - 2002-01-07 09:28:47
|
Hello, ----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> To: "OpenDemo Development List" <ope...@li...> Sent: Monday, January 07, 2002 1:22 AM Subject: Re: [opendemo-devel] Perfomance. > [...] > A ring-buffer would simply solve this bottleneck of constantly calling > fbGrow(), mbResize and memcpy() but we have 21 occurrences of mbGetBuffer(), > whose are not so easily converted to a ring-buffer. We need to know before > calling mbGetBuffer(), how many bytes should be available. Maybe this is > already done with the usually fbGrow() call before mbGetBuffer() but I'm not > really sure in all 21 cases. If you put the ring-buffer code in od_filebuf.c instead of od_membuf.c (or add a middle layer), then that shouldn't be a problem. The fb* interface does not provide direct access to the buffer, so changing from linear to ring buffers shouldn't require other changes. mbGetBuffer() just gives you the start of the memory block (same as any other ring-buffer). -Conor |
|
From: Dr. U. G. <Uwe...@ph...> - 2002-01-07 07:22:00
|
Hello! > > ... Please perform another profiler run first. > Gotcha, done them. > I've had to redo the profiling with a special demo for this (my demo i've > run previous time was lost), the demo spans for 20 seconds and is uploaded > to cvs (1 megabyte) with a config and windows bat file, so if you set up the > linux profiling, use the config which starts the demo, they're still not > perfect, because after the demo ends server doesn't quit automatically and > library isn't unloaded, thus making stale calls to vmmain and affecting the > stats a bit. 2 problems here: I tried the linux profiler gprof (together with the GCC compiler option -pg) but this somehow gave me no result file. I suspect, that a special initialization function will be called before main() in such binaries but we link only shared libraries, so this function call is missing. I tried the environment varibale LD_PROFILE, which was made exactly for the purpose of profiling a shared library but the ld.so included profiler had problems loading the nvidia OpenGL driver (I did not try to profile this library but it is somehow involved too (at least for time measurements)). So I need to find the init-function, call it myself in vmMain() and'll try -pg/gprof again. > >From what I can see, the optimizations are clearly viisble, even without > profiling my fps has gone better. These are good news. > Func Func+Child Hit > Time % Time % Count Function > --------------------------------------------------------- > 852,447 15,8 867,984 16,1 40371 _odpStartElement (odp_parse.obj) This comes from the object memcopy. Beside _odpUpdateEntity and _xmlParseName the next functions show all only symptons of the same problem: the linear buffer, which is a direct copy of a part of the file. As Conor already mentioned, a ring-buffer would be much better here. I'll think about it (and I have already coded several ring-buffers in previous projects). > 706,381 13,1 1925,211 35,7 2791160 _fbGrow (od_filebuf.obj) > 602,798 11,2 602,798 11,2 1457110 _trap_FS_Read (g_syscalls.obj) > 470,155 8,7 2291,271 42,5 2037568 _fbReadChar (od_filebuf.obj) > 441,877 8,2 1044,675 19,4 1457110 _odFread (od_q3a_fileio.obj) > 244,663 4,5 394,574 7,3 404480 _odpUpdateEntity (odp_main.obj) > 224,653 4,2 224,653 4,2 1768386 _mbResize (od_membuf.obj) C library realloc() is out of reach for profiling, so we get the same result for Func and Func+Child > 198,653 3,7 1667,464 30,9 110023 _xmlParseName (od_xml_read.obj) > 160,243 3,0 241,820 4,5 643569 _fbSkip (od_filebuf.obj) A ring-buffer would simply solve this bottleneck of constantly calling fbGrow(), mbResize and memcpy() but we have 21 occurrences of mbGetBuffer(), whose are not so easily converted to a ring-buffer. We need to know before calling mbGetBuffer(), how many bytes should be available. Maybe this is already done with the usually fbGrow() call before mbGetBuffer() but I'm not really sure in all 21 cases. The second problem are the new files in the main directory! Eugene, please move them into a newly created profiling (or test/profile or test/profile/data and test/profile/bin) directory. The main directory is really not the place for this kind of files. Bye, Uwe |
|
From: Eugene 'H. B. <hm...@ma...> - 2002-01-06 18:50:50
|
----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> Subject: Re: [opendemo-devel] Perfomance. > > I think we should collect the characters in a local variable and simply copy > > them over after odMalloc(). fbRead() calls fbGrow() and 2 times mbSize(), > > no wonder that it takes so much time. > This is not done yet. Please perform another profiler run first. Gotcha, done them. I've had to redo the profiling with a special demo for this (my demo i've run previous time was lost), the demo spans for 20 seconds and is uploaded to cvs (1 megabyte) with a config and windows bat file, so if you set up the linux profiling, use the config which starts the demo, they're still not perfect, because after the demo ends server doesn't quit automatically and library isn't unloaded, thus making stale calls to vmmain and affecting the stats a bit. I have attached result files, "odprofile1.txt" is the unoptimized source re-run, and "odprofile2.txt" is with recent optimizations. Files that uwe modified recently were downgraded to previous versions for re-profiling unoptimized code (od_membuf.[ch] and od_xml_read.c)... This will give a nice chart with the modifications we get. From what I can see, the optimizations are clearly viisble, even without profiling my fps has gone better. |
|
From: Eugene 'H. B. <hm...@ma...> - 2002-01-05 17:51:05
|
----- Original Message ----- From: "Dr. Uwe Girlich" <Uwe...@ph...> Subject: [opendemo-devel] download statistics > Has anyone looked at the "Usage Statistics" page for OpenDemo? > but the download rate is really low. Maybe we should also offer zip files Good idea, and also it's a good idea putting these three packages into one only for zips. > The second point is the 25th of december. What happended then? We've got over > 120 page views and even more downloads! I suspect some kind of web robot. > Any ideas? There were news posts on clanbase.com and cyberfight.ru sites around 25th of december. Clanbase was the first to spot (first after machinima) -------- - Eugene 'HMage' Bujak |
