Menu
▾
▴
civlog-list — List about general CivLog topics
You can subscribe to this list here.
| 2000 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(97) |
Oct
(16) |
Nov
|
Dec
|
|---|
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 15:08:49
|
I added some measurement code to Graph:drawGraph() and got the following accumulated (since the induvidual times were to small) times: :29ms for setup :5243ms for adding points :907ms for renderGroup :2451ms for drawing :9689ms overall This is done using a PolyLine which uses Graphics.drawPolyline() instead of multiple Graphics.drawLines(). I'm impressed by the good number for renderGroup. I expect the number for "adding points" will improve if the data structures were changed. Raimar -- A supercomputer is a computer running an endless loop in just a second |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 15:03:25
|
On Mon, Sep 25, 2000 at 10:32:33PM +0800, Wong TM (Huang Deming) wrote:
>
> Changes:
>
> * Ulrich have made a lot of changes to restructure the data structure:
> - .......wait till I read through the mail.
I think you made the inclusion a bit to fast. The work of Ulrich has made
aware some questions wrt Score. We now think we how to do it the "right way
[tm]" so wait for a final version which should come in the next days.
Raimar
--
"Many of my assistants were fans of Tolkien, who wrote 'Lord of the Rings'
and a number of other children's stories for adults. The first character
alphabet that was programmed for my plotter was Elvish rather than Latin."
-- from SAIs "life as a computer for a quarter of a century"
|
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-25 14:32:49
|
Just a quick FYI, use 'make indent' before submitting files, astyle is needed http://astyle.sourceforge.net/. I added a new make target 'indent2' for yours' and Michael style. @see next post. -Deming ----------------------------- /"\ ASCII-Ribbon Campaign \ / """""""""""""""""""" x No HTML or WORD in Mails / \ HTML is for WEB, Word is for Micro$oft. ----------------------------- |
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-25 14:32:22
|
Changes:
* Ulrich have made a lot of changes to restructure the data structure:
- .......wait till I read through the mail.
-
* Add Raimar comment on log files added into the Readme.
* Michael patch applied. Add an dialog box to info user of loading error.
* One new make target: 'archive' use to make a tarball
* One new make target: 'indent2', same as <indent> but produce source with format
with 'attach bracket'(java style)
Thank you Ulrich and of course Raimar and Michael providing feedback to Ulrich. Keep
up the good work!
Deming
-----------------------------
/"\ ASCII-Ribbon Campaign
\ / """"""""""""""""""""
x No HTML or WORD in Mails
/ \ HTML is for WEB, Word is
for Micro$oft.
-----------------------------
|
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-25 14:31:37
|
> -----Original Message-----
> From: civ...@li...
> [mailto:civ...@li...]On Behalf Of Ulrich Kuhn
>
> On Mon, 25 Sep 2000, Wong TM (Huang Deming) wrote:
>
> > > -----Original Message-----
> > > From: civ...@li...
> > > [mailto:civ...@li...]On Behalf Of Raimar
> > > Falke
> > >
> > > Have you tried GIF or PNG?
> >
> > FYI. JPG use 'lossy' compression(therefore the noise around the text).
> > Therefore jpg files is always smaller than GIF files which use 'loseless'
> > compression(better quality). AFAIK PNG is in the GIF family?
> >
> > Oh! You mean GIF use 256 color therefore smaller. I tried it and it is
> > much smaller. CivLog doesn't used much color therefore good with GIF.
> > Will update the pics. tomorrow.
>
> I made a little comparison:
> Screenshot of my CivLog-window in
> JPG 90KB (quality level: 0.85)
> GIF 35KB (quantized to 32 colors)
> PNG 35KB (truecolor)
>
> I think we should use PNG because it supports truecolor, has a good
> (lossless) compression and uses no patented compression algorithm (as GIF
> does).
What?! PNG use some black magic or what?! It lossless, truecolor, opensource and
compress real small, it's perfect!
-Deming
-----------------------------
/"\ ASCII-Ribbon Campaign
\ / """"""""""""""""""""
x No HTML or WORD in Mails
/ \ HTML is for WEB, Word is
for Micro$oft.
-----------------------------
|
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 12:18:10
|
On Mon, Sep 25, 2000 at 01:33:02PM +0100, Michael Grundel wrote:
> Raimar wrote:
> > On Mon, Sep 25, 2000 at 12:54:53AM +0100, Michael Grundel wrote:
> > > > On Sun, 24 Sep 2000, Michael Grundel wrote:
> > > > >
> > > > > Ok, I just had a brief look at the code and I think your are making a wrong
> > > > > assumption on the number of players. The number of players is not const!
> > > >
> > > > I made an quick work-around by ignoring any player number "beyond" the
> > > > valid range. Maybe this is even a final solution because these additional
> > > > players have no name and how should one display them then?
> > >
> > > I think that is the way it currently is handled (at least they are not
> > > displayed).
> >
> > Which is not good but Ulrich's code is worse.
> >
> [snip discussion about mem usage, etc]
> > > But I'd like to point out that it is the runtime speed I am concerned about,
> > > not the memory.
> >
> > Ok after sleeping a night about this problem I think I found the ideal
> > solution if the following conditions hold:
> > - player, data types and turns are continuous starting from 0
> > - we know the number of players, data types and turns in advance (see below
> > for a solution)
>
> I agree with the described solution below. It's simple and efficient and IMHO
> we should do it that way.
>
> Maybe we can discuss some minor implementation details.
> I noticed that for some reason (not sure why) the player number is not
> always continuous. I think it may be a bug in freeciv.
> I have more than one logfile like this (note the missing player 27):
> ...
> 26 24 -225 10
> 26 25 -225 8
> 26 26 -225 25
> 0 0 -200 57
> 0 1 -200 64
> 0 2 -200 92
> 0 3 -200 59
> 0 4 -200 35
> 0 5 -200 12
> 0 6 -200 60
> 0 7 -200 112
> 0 8 -200 99
> 0 9 -200 35
> 0 10 -200 71
> 0 11 -200 151
> 0 12 -200 75
> 0 13 -200 93
> 0 14 -200 56
> 0 15 -200 52
> 0 16 -200 43
> 0 17 -200 64
> 0 18 -200 72
> 0 19 -200 48
> 0 20 -200 40
> 0 21 -200 46
> 0 22 -200 40
> 0 23 -200 115
> 0 24 -200 51
> 0 25 -200 40
> 0 26 -200 63
> 0 28 -200 28 <----- note the missing player 27
> ...
> It only seems to happen when additional players are introduced. I guess it
> would be easier to fix freeciv than to have our algo deal with it? It's
> probably just a "<" vs. "<=" issue.
I didn't notice this problem. Maybe we should catch access to unset data in
our internal structures.
> > class Score
> > {
> > ...
> > private DataTypeInfo dataTypeInfos[]; // indexed by data type
> > ...
> > }
> >
> > class DataTypeInfo
> > {
> > ...
> > private DataTypeInfoForOneTurn data[]; // indexed by turn
> > ...
> > }
> >
> > class DataTypeInfoForOneTurn
> > {
> > ...
> > private int values[]; // values indexed by player
> > ...
> > }
> >
> > - this solution is a 3D array indexed by data type, turn and player
> > - it should provide the fastes access
> > - it should provide the lowest memory consumption (not exactly it may be
> > possible to use something like
> > class Score
> > {
> > private int data[];
> > public int getValue(int dataType, int turn, int player)
> > {
> > return data[dataType*turns*players+turn*players+player];
> > }
> > ....
> > data=new int[dataTypes*turns*players];
> > ....
> > }
> > )
> >
> > - it allows the Graph to cache some of the indexing
>
> Both solutions are fine with me.
> I'd favor the 3 individual arrays. They are better suited for use in Graph
> because caching is more straight-forward.
Nod.
>
> > It was not possible to use such a scheme before since the old way of
> > indexing the data using the year value would made the array sparse (and with
> > a negative index but this is only a minor problem)
> >
> > Since we allocate all arrays before we read the main part of the score file
> > we have to know the dimensions.
>
> That would make the storage-classes easier but the parsing more troublesome.
> Alternatively we could implement the storage-classes to be growable.
> e.g.
> class DataTypeInfo
> {
> ...
> private DataTypeInfoForOneTurn data[]; // indexed by turn
>
> public grow(int turn) {
> if (turn >= data.length) {
> // create bigger array and copy the values
> DataTypeInfoForOneTurn data_new[] = new DataTypeInfoForOneTurn [data.length+20];
> System.arraycopy(data, 0, data_new, 0, data.length);
>
> // init the grown values
> for (int i = data.length; i < data_new.length; i++)
> data_new[i] = new DataTypeInfoForOneTurn();
>
> data = data_new;
> System.gc(); // garbage collect, not really necessary
> }
> }
> ...
> }
>
> and in Score when adding a value:
> try {
> dataTypeInfos[type][turn][player] = value; // try to add value
> } catch (ArrayIndexOutOfBoundsException ex) {
> dataTypeInfos[type].grow(turn); // if overflow -> grow and init
> dataTypeInfos[type][turn].grow(player);
> dataTypeInfos[type][turn][player] = value; // when growing is complete add value
> }
>
> Hm, after writing this, it looks more complicated than I thought.
> Maybe it's easier to just parse the log to get the max-values as you
> suggest.
We have to do some timing measurements.
> > We learned that players exists which
> > aren't "declared" in the player section first. This problem can be solved by
> > looking at the last line which hold the last player index.
>
> yes
>
> > A greater problem is the number of turns. I see tow possible solutions:
> > - eastimate from the final year the number of turn (this can't be accurate
> > since the mapping changes)
>
> I don't know the formula, but maybe it can be computed based on the first
> year and the last year?
No. From freeciv's common/game.c:game_next_year()
spaceshipparts= 0;
if (game.spacerace) {
for(i=0; parts[i] < B_LAST; i++) {
int t = improvement_types[parts[i]].tech_req;
if(tech_exists(t) && game.global_advances[t])
spaceshipparts++;
}
}
if( year >= 1900 || ( spaceshipparts>=3 && year>0 ) )
year += 1;
else if( year >= 1750 || spaceshipparts>=2 )
year += 2;
else if( year >= 1500 || spaceshipparts>=1 )
year += 5;
else if( year >= 1000 )
year += 10;
else if( year >= 0 )
year += 20;
else if( year >= -1000 ) /* used this line for tuning (was -1250) */
year += 25;
else
year += 50;
if (year == 0)
year = 1;
It depends on the global advance measured in space ship parts.
Raimar
--
"Heuer's Law: Any feature is a bug unless it can be turned off."
|
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 12:04:19
|
On Mon, Sep 25, 2000 at 12:56:35PM +0200, Raimar Falke wrote: > > > One thing about firstTurn and lastTurn in Score.java: Is it possible that > > firstTurn is unequal to 0? I think the first turn in a log file is always > > 0. Only startYear and endYear vary. So countTurns() should be enough > > access for turns in Score.java. > > We have to consider multiple score files. The following is a copy-and-paste > of an old mail: > > -- start -- > > > - multiple scorefiles which are put together by civlog > > > > Some parsing and merging. Had to modified freeciv(or a script) logging function > > to output civscore(0-9).log rather than overwriting civscore.log everytime. Merge > > civscore*.log together judging by the same players tag, sorted by years > > (timestamp of files would be easier but not reliable) > > We copy Score to SingleScore. Clean Score and make it an abstract class. > Create MultipleScore. MultipleScore will get a list of filenames or Scores > and will multiplex them. No interface change, clean seperation. And we have > to make the changes to freeciv you mentioned. > -- end -- > > [ I read it again and the english is hard to understand ] Since we change Score we can also incorporate this with the proposal above. I attached the interface Score and the class NullScore. NullScore does common things for both SimpleScore and MultipleScore. Our comments are welcome. All code is untested. So even in this splitting of the code firstTurn would hardcoded to 0. Raimar -- What's nice about GUI is that you see what you manipulate. What's bad about GUI is that you can only manipulate what you see. |
|
From: Michael G. <mic...@gr...> - 2000-09-25 11:32:26
|
Raimar wrote:
> On Mon, Sep 25, 2000 at 12:54:53AM +0100, Michael Grundel wrote:
> > > On Sun, 24 Sep 2000, Michael Grundel wrote:
> > > >
> > > > Ok, I just had a brief look at the code and I think your are making a wrong
> > > > assumption on the number of players. The number of players is not const!
> > >
> > > I made an quick work-around by ignoring any player number "beyond" the
> > > valid range. Maybe this is even a final solution because these additional
> > > players have no name and how should one display them then?
> >
> > I think that is the way it currently is handled (at least they are not
> > displayed).
>
> Which is not good but Ulrich's code is worse.
>
[snip discussion about mem usage, etc]
> > But I'd like to point out that it is the runtime speed I am concerned about,
> > not the memory.
>
> Ok after sleeping a night about this problem I think I found the ideal
> solution if the following conditions hold:
> - player, data types and turns are continuous starting from 0
> - we know the number of players, data types and turns in advance (see below
> for a solution)
I agree with the described solution below. It's simple and efficient and IMHO
we should do it that way.
Maybe we can discuss some minor implementation details.
I noticed that for some reason (not sure why) the player number is not
always continuous. I think it may be a bug in freeciv.
I have more than one logfile like this (note the missing player 27):
...
26 24 -225 10
26 25 -225 8
26 26 -225 25
0 0 -200 57
0 1 -200 64
0 2 -200 92
0 3 -200 59
0 4 -200 35
0 5 -200 12
0 6 -200 60
0 7 -200 112
0 8 -200 99
0 9 -200 35
0 10 -200 71
0 11 -200 151
0 12 -200 75
0 13 -200 93
0 14 -200 56
0 15 -200 52
0 16 -200 43
0 17 -200 64
0 18 -200 72
0 19 -200 48
0 20 -200 40
0 21 -200 46
0 22 -200 40
0 23 -200 115
0 24 -200 51
0 25 -200 40
0 26 -200 63
0 28 -200 28 <----- note the missing player 27
...
It only seems to happen when additional players are introduced. I guess it
would be easier to fix freeciv than to have our algo deal with it? It's
probably just a "<" vs. "<=" issue.
> class Score
> {
> ...
> private DataTypeInfo dataTypeInfos[]; // indexed by data type
> ...
> }
>
> class DataTypeInfo
> {
> ...
> private DataTypeInfoForOneTurn data[]; // indexed by turn
> ...
> }
>
> class DataTypeInfoForOneTurn
> {
> ...
> private int values[]; // values indexed by player
> ...
> }
>
> - this solution is a 3D array indexed by data type, turn and player
> - it should provide the fastes access
> - it should provide the lowest memory consumption (not exactly it may be
> possible to use something like
> class Score
> {
> private int data[];
> public int getValue(int dataType, int turn, int player)
> {
> return data[dataType*turns*players+turn*players+player];
> }
> ....
> data=new int[dataTypes*turns*players];
> ....
> }
> )
>
> - it allows the Graph to cache some of the indexing
Both solutions are fine with me.
I'd favor the 3 individual arrays. They are better suited for use in Graph
because caching is more straight-forward.
> It was not possible to use such a scheme before since the old way of
> indexing the data using the year value would made the array sparse (and with
> a negative index but this is only a minor problem)
>
> Since we allocate all arrays before we read the main part of the score file
> we have to know the dimensions.
That would make the storage-classes easier but the parsing more troublesome.
Alternatively we could implement the storage-classes to be growable.
e.g.
class DataTypeInfo
{
...
private DataTypeInfoForOneTurn data[]; // indexed by turn
public grow(int turn) {
if (turn >= data.length) {
// create bigger array and copy the values
DataTypeInfoForOneTurn data_new[] = new DataTypeInfoForOneTurn [data.length+20];
System.arraycopy(data, 0, data_new, 0, data.length);
// init the grown values
for (int i = data.length; i < data_new.length; i++)
data_new[i] = new DataTypeInfoForOneTurn();
data = data_new;
System.gc(); // garbage collect, not really necessary
}
}
...
}
and in Score when adding a value:
try {
dataTypeInfos[type][turn][player] = value; // try to add value
} catch (ArrayIndexOutOfBoundsException ex) {
dataTypeInfos[type].grow(turn); // if overflow -> grow and init
dataTypeInfos[type][turn].grow(player);
dataTypeInfos[type][turn][player] = value; // when growing is complete add value
}
Hm, after writing this, it looks more complicated than I thought.
Maybe it's easier to just parse the log to get the max-values as you
suggest.
> We learned that players exists which
> aren't "declared" in the player section first. This problem can be solved by
> looking at the last line which hold the last player index.
yes
> A greater problem is the number of turns. I see tow possible solutions:
> - eastimate from the final year the number of turn (this can't be accurate
> since the mapping changes)
I don't know the formula, but maybe it can be computed based on the first
year and the last year?
> - count the turns by doing an extra loop over the lines:
> for(int lineno=0;lineno<lines.length;lineno++)
> {
> if(lines[lineno].startsWith("0 0 "))
> turn++;
> }
> - adjust freeciv to report the year and the turn
>
> > Ahh, so the memory consumption comes from loading the score file and not
> > only from the data-structure.
> > If that is so, I would like to suggest that we keep the old data-structure
> > and the new loadScoreFile-method. Would that be possible?
> >
> > Maybe it's only me, but I find the new data-structure hard to understand
> > (not in principle but in the code), so I would vote for the simpler model.
>
> I support you.
Fine, I think your solution is the best so far. I hope Ulrich agrees that
this structure is easier and as memory-saving as his.
> > > > > > - the whole score thing should be using turns instead of years as primary
> > > > > > reference. At the beginning everything used years than the GUI was
> > > > > > changed to use turns and turn2year and year2turn was added. It
> > > > > > would now be time to change score to also use turns.
> >
> > If we do a change to the data-structures, I would vote for changing it
> > in that direction rather than all that complicated stuff now being introduced.
> > It makes data-storage even easier: array indexed by turn (see above).
> > BTW, who wrote that?
>
> I wrote this and very first version (you remember). As I wrote in a previous
> mail it is now time to adjust Score also to use turns. I think I choose year
> at the first because the turn number is not available for free. (it is only
> a small calculation but nevertheless)
Ok, I guess I skipped some messages on the list.
Michael
--
eMail: mic...@gr...
PGP key available on request
|
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 10:56:38
|
On Mon, Sep 25, 2000 at 12:34:54PM +0200, Ulrich Kuhn wrote:
>
> HELP! There's a anti Ulrich's code union (AUCU) forming! I'll have to
> emigrate to Switzerland and eat Käsefondue all day! ;-)
No. It is just that you bring some of the problems with Score into the
light.
> I have made all public variables in Score.java private and provided access
> methods. As a consequence I changed Graph.java, RangeMonitor.java and
> MainFrame.java to invoke these methods.
Nice.
> I added CounterInputStrea.java (in util) to simplify the usage of a
> ProgressMonitor during loading and parsing in Score.loadScoreFile(). So
> there's no need anymore for storing a String object for every line. Lines
> just can be read, parsed and then discarded.
Please take a look at javax.swing.ProgressMonitorInputStream.
> What has to be done?
>
> Changing the data structure. Especially indexing data by turn and not by
> year. Score has currently a method
> public int lookupValue(int player, int type, int year);
> it could be changed to
> public int lookupValue(int player, int type, int turn);
> But as a consequence Graph.java has to be majorly restructured as well.
If we change the structure like I mentioned in the last email we need
something like:
public DataTypeInfo getDataTypeInfo(int dataType);
private void setData(int player, int dataType, int turn, int value);
> One thing about firstTurn and lastTurn in Score.java: Is it possible that
> firstTurn is unequal to 0? I think the first turn in a log file is always
> 0. Only startYear and endYear vary. So countTurns() should be enough
> access for turns in Score.java.
We have to consider multiple score files. The following is a copy-and-paste
of an old mail:
-- start --
> > - multiple scorefiles which are put together by civlog
>
> Some parsing and merging. Had to modified freeciv(or a script) logging function
> to output civscore(0-9).log rather than overwriting civscore.log everytime. Merge
> civscore*.log together judging by the same players tag, sorted by years
> (timestamp of files would be easier but not reliable)
We copy Score to SingleScore. Clean Score and make it an abstract class.
Create MultipleScore. MultipleScore will get a list of filenames or Scores
and will multiplex them. No interface change, clean seperation. And we have
to make the changes to freeciv you mentioned.
-- end --
[ I read it again and the english is hard to understand ]
>
> One thing about memory measurements: If you want to find out how many
> (java internal) memory an object or a group of them use you can do the
> following thing in java
> System.gc();
> long mem1 = Runtime.totalMemory()-Runtime.freeMemory();
> // instantiate your object
> long mem2 = Runtime.totalMemory()-Runtime.freeMemory();
> the difference mem2-mem1 would than be the memory consumption. But be
> careful this is only precise in the range of hundreds of kilo bytes. So
> you cannot measure the memory consumption of one String object but 10000
> objects can be measured.
> There are four methods at the end of Score.java showing you this.
Nod.
> Outlook: I will hapily change the data structure for Score.java. But
> somebody other has to adapt Graph.java.
No problem I will do it.
> I hope this was enough for now. Maybe the first- and lastTurn thing splits
> the AUCU in rivaling parts fighting themselves. ;-)
*grin*
Raimar
--
The Software is not designed or licensed for use in on-line control
equipment in hazardous environments, such as operation of nuclear
facilities, aircraft navigation or control, or direct life support
machines.
-- Java Compiler Compiler Download and License Agreement
|
|
From: Ulrich K. <ku...@un...> - 2000-09-25 10:34:57
|
HELP! There's a anti Ulrich's code union (AUCU) forming! I'll have to emigrate to Switzerland and eat Käsefondue all day! ;-) More serious: I support the request of the AUCU of restructuring the data structure used by Score.java. I also support the idea of reusing existing java api code but only if it isn't too slow and not using too much memory. I think the usage of arrays as Raimar mentioned in his last email for storing the data is a good thing (indexed by player number, turn and data type). More detailed: I have attached my recent code changes. They do not reflect the above points yet. But they contain all the earlier mentioned "bug fixes". What has changed so far? I have made all public variables in Score.java private and provided access methods. As a consequence I changed Graph.java, RangeMonitor.java and MainFrame.java to invoke these methods. I added CounterInputStrea.java (in util) to simplify the usage of a ProgressMonitor during loading and parsing in Score.loadScoreFile(). So there's no need anymore for storing a String object for every line. Lines just can be read, parsed and then discarded. What has to be done? Changing the data structure. Especially indexing data by turn and not by year. Score has currently a method public int lookupValue(int player, int type, int year); it could be changed to public int lookupValue(int player, int type, int turn); But as a consequence Graph.java has to be majorly restructured as well. One thing about firstTurn and lastTurn in Score.java: Is it possible that firstTurn is unequal to 0? I think the first turn in a log file is always 0. Only startYear and endYear vary. So countTurns() should be enough access for turns in Score.java. One thing about memory measurements: If you want to find out how many (java internal) memory an object or a group of them use you can do the following thing in java System.gc(); long mem1 = Runtime.totalMemory()-Runtime.freeMemory(); // instantiate your object long mem2 = Runtime.totalMemory()-Runtime.freeMemory(); the difference mem2-mem1 would than be the memory consumption. But be careful this is only precise in the range of hundreds of kilo bytes. So you cannot measure the memory consumption of one String object but 10000 objects can be measured. There are four methods at the end of Score.java showing you this. Just out of curiosity I made an comparison between searching in an array and looking up the same value in a hash table: Using a hashtable was four times slower. But wait theres more: the array was a int[] thing and the hash table used objects. These objects had to be created before every lookup. My conclusion is this: If you've got objects you want to store in a hash table -> do it. If you have no objects but int values -> store them in arrays. I have used a hash table in Score.java to store mappings between data type texts and their int index value in the log file. Outlook: I will hapily change the data structure for Score.java. But somebody other has to adapt Graph.java. I hope this was enough for now. Maybe the first- and lastTurn thing splits the AUCU in rivaling parts fighting themselves. ;-) Best regards Ulrich (Damn! Where have I put my asbestos suit?!) |
|
From: Ulrich K. <ku...@un...> - 2000-09-25 09:38:44
|
On Sun, 24 Sep 2000, Raimar Falke wrote: > On Sun, Sep 24, 2000 at 10:41:27PM +0800, Wong TM (Huang Deming) wrote: > > </rubbish>*@!? Windoze, outlook cause page fault while editing....!@$@#$% > > $$$$$$sucker@#$#@$ </rubbish> > > > -----Original Message----- > > > From: civ...@li... > > > [mailto:civ...@li...]On Behalf Of Raimar > > > Falke > > > > > > On Sun, Sep 24, 2000 at 10:37:43AM +0800, Wong TM (Huang Deming) wrote: > > > > > > > [ AFAIK freeciv-dev would be a better place to such a proposal ] > > > It is also possible to omit lines if the value didn't change. > > > Or it may be > > > possible to compress the file. It looks like this is used for > > > the network > > > protocol and so zlib is a requirement. Nevertheless all these changes > > > complicate thing during the parsing. I don't know how many > > > programs are our > > > there which can parse the current format and will then need an update. > > > > At least quicker loading for smaller log. > > I think the current format can be parsed straightforward. This is a big > advantage. We may look at our reading method and try to see what make it > slow (reading the file, parsing, creating internal data structures, setting > data in the internal data structures) It should be easy to comment some > parts out and measure the performance. I can also imagine a small C program > which parses the scorefile and produce a file in an interim format which can > be understand more easily be the java program. C <shiver> Compression with the zlib algorithm would be not a great deal with java: Since version 1.1 theres the java.util.zip package which provides support for compressing and decompressing data. BTW: These are the same routines used by jar to create and compress (and of course decompress) jar files. But I also tend to having an umcompressed file with just the plain values. You mustn't transfer log files often over the net so the size of the log files should be no big problem. Or, well, maybe size is a problem. Err. Ok, I'm undecided. I have nothing against compression in any useful form. Ulrich |
|
From: Ulrich K. <ku...@un...> - 2000-09-25 09:29:47
|
On Mon, 25 Sep 2000, Wong TM (Huang Deming) wrote: > > -----Original Message----- > > From: civ...@li... > > [mailto:civ...@li...]On Behalf Of Raimar > > Falke > > > > Have you tried GIF or PNG? > > FYI. JPG use 'lossy' compression(therefore the noise around the text). > Therefore jpg files is always smaller than GIF files which use 'loseless' > compression(better quality). AFAIK PNG is in the GIF family? > > Oh! You mean GIF use 256 color therefore smaller. I tried it and it is > much smaller. CivLog doesn't used much color therefore good with GIF. > Will update the pics. tomorrow. I made a little comparison: Screenshot of my CivLog-window in JPG 90KB (quality level: 0.85) GIF 35KB (quantized to 32 colors) PNG 35KB (truecolor) I think we should use PNG because it supports truecolor, has a good (lossless) compression and uses no patented compression algorithm (as GIF does). BTW: The compression used by PNG is zlib. So it could be easily en- and decoded by java (see next mail). Sincerely Ulrich |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-25 09:18:48
|
On Mon, Sep 25, 2000 at 12:54:53AM +0100, Michael Grundel wrote:
> > Tomorrow I will send in all the modifications I'll mention here and
> > had earlier.
> >
> > On Sun, 24 Sep 2000, Michael Grundel wrote:
> >
> > > > > > Please send comments.
> > >
> > > I really appreciate your contribution. But it seems to be very slow.
> > >
> > > There also is a bug that results in a
> > > java.lang.ArrayIndexOutOfBoundsException
> > > at org.freeciv.civlog.data.Score.loadScoreFile(Score.java:353)
> > > at org.freeciv.civlog.gui.MainFrame.openFile(MainFrame.java:480)
> > >
> > > It affects almost all of my logfiles. It is the same one that is mentioned
> > > by Deming.
> > >
> > > Ok, I just had a brief look at the code and I think your are making a wrong
> > > assumption on the number of players. The number of players is not const!
> >
> > I made an quick work-around by ignoring any player number "beyond" the
> > valid range. Maybe this is even a final solution because these additional
> > players have no name and how should one display them then?
>
> I think that is the way it currently is handled (at least they are not
> displayed).
Which is not good but Ulrich's code is worse.
> > > While I was at it, I did a quick calc on mem usage of your old solution:
> > > Our hash (player, turn) -> (ScoreEntry) would take about:
> > > 20 players * 500 turns * 50 bytes = 500 KBytes of memory
> >
> > For every player and every turn you have a Hashtable
> > (ScoreEntry) containing for each data type two Integers (the key and the
> > value).
> > 20 player * 500 turns * (100 + 40*50) = 21 MByte
> > Hashtable ^ ^ Integers
>
> Huh?
> ScoreEntry has a Hash? OOps it is, but it should not.
> Of course ScoreEntry is not really part of the lookup-hash-structure.
> I admit it should not use a hash at all (it is a legacy from a time
> when information about data-types was not available).
> You can see that it can easily be changed to use a simple array with
> index = data_type. I have attached the modified file. I'd like to hear
> about it's memory consumption on your system (because it seems to differ
> from mine).
>
> On my system, it decreases the memory requirement of the superbig.log from
> the website from 36 MB to 26 MB. Now my above calc should be correct, yes?
> I only count the overhead for the hash lookup infrastructure, because the
> real data is now in arrays.
>
> But I'd like to point out that it is the runtime speed I am concerned about,
> not the memory.
Ok after sleeping a night about this problem I think I found the ideal
solution if the following conditions hold:
- player, data types and turns are continuous starting from 0
- we know the number of players, data types and turns in advance (see below
for a solution)
class Score
{
...
private DataTypeInfo dataTypeInfos[]; // indexed by data type
...
}
class DataTypeInfo
{
...
private DataTypeInfoForOneTurn data[]; // indexed by turn
...
}
class DataTypeInfoForOneTurn
{
...
private int values[]; // values indexed by player
...
}
- this solution is a 3D array indexed by data type, turn and player
- it should provide the fastes access
- it should provide the lowest memory consumption (not exactly it may be
possible to use something like
class Score
{
private int data[];
public int getValue(int dataType, int turn, int player)
{
return data[dataType*turns*players+turn*players+player];
}
....
data=new int[dataTypes*turns*players];
....
}
)
- it allows the Graph to cache some of the indexing
It was not possible to use such a scheme before since the old way of
indexing the data using the year value would made the array sparse (and with
a negative index but this is only a minor problem)
Since we allocate all arrays before we read the main part of the score file
we have to know the dimensions. We learned that players exists which
aren't "declared" in the player section first. This problem can be solved by
looking at the last line which hold the last player index.
A greater problem is the number of turns. I see tow possible solutions:
- eastimate from the final year the number of turn (this can't be accurate
since the mapping changes)
- count the turns by doing an extra loop over the lines:
for(int lineno=0;lineno<lines.length;lineno++)
{
if(lines[lineno].startsWith("0 0 "))
turn++;
}
- adjust freeciv to report the year and the turn
> Ahh, so the memory consumption comes from loading the score file and not
> only from the data-structure.
> If that is so, I would like to suggest that we keep the old data-structure
> and the new loadScoreFile-method. Would that be possible?
>
> Maybe it's only me, but I find the new data-structure hard to understand
> (not in principle but in the code), so I would vote for the simpler model.
I support you.
> > > > > - the whole score thing should be using turns instead of years as primary
> > > > > reference. At the beginning everything used years than the GUI was
> > > > > changed to use turns and turn2year and year2turn was added. It
> > > > > would now be time to change score to also use turns.
>
> If we do a change to the data-structures, I would vote for changing it
> in that direction rather than all that complicated stuff now being introduced.
> It makes data-storage even easier: array indexed by turn (see above).
> BTW, who wrote that?
I wrote this and very first version (you remember). As I wrote in a previous
mail it is now time to adjust Score also to use turns. I think I choose year
at the first because the turn number is not available for free. (it is only
a small calculation but nevertheless)
Raimar
--
This message has been ROT-13 encrypted twice for extra security.
|
|
From: Michael G. <mic...@gr...> - 2000-09-24 22:55:07
|
> Tomorrow I will send in all the modifications I'll mention here and > had earlier. > > On Sun, 24 Sep 2000, Michael Grundel wrote: > > > > > > Please send comments. > > > > I really appreciate your contribution. But it seems to be very slow. > > > > There also is a bug that results in a > > java.lang.ArrayIndexOutOfBoundsException > > at org.freeciv.civlog.data.Score.loadScoreFile(Score.java:353) > > at org.freeciv.civlog.gui.MainFrame.openFile(MainFrame.java:480) > > > > It affects almost all of my logfiles. It is the same one that is mentioned > > by Deming. > > > > Ok, I just had a brief look at the code and I think your are making a wrong > > assumption on the number of players. The number of players is not const! > > I made an quick work-around by ignoring any player number "beyond" the > valid range. Maybe this is even a final solution because these additional > players have no name and how should one display them then? I think that is the way it currently is handled (at least they are not displayed). > > > > - why do you use arrays in YearValueMap instead of a hash? Have you made > > > > some measurements? > > > > > > Not needed: > > > > Performance measurements are needed (it feels so slow). > > > > > http://www.javaworld.com/javaworld/jw-11-1999/jw-11-performance-2.html > > > > > > In short: (java) objects needs (besides the raw values) some memory to do > > > > Arrays.... now I understand why it is so slow :) > > Why do you think arrays are slow? I had explained it further down. I was referring to search (of course if it's ordered you get O(log N)). > > To explain our existing data-structure: > > So far our efforts were never concerned with mem usage but with execution > > speed. > > > > To be honest, I think your implementation is too slow. I tried a relatively > > small logfile with only 5 players and even with that one it is too slow for > > my taste. > > I cannot test it with a big logfile because the bug prevents me from loading > > them. > > According to my testing (with top under Linux and java internal) "my > solution" is two times faster and uses the fifth part of memory than the > "old solution". Interesting, I cannot confirm this. On my OS it is a little better on memory (23 to 19 MB) and feels very slow. What exactly is it doing faster? Ok, how did I "perceive" the speed? I noticed it takes a long time to add additional data-types (I usually use 3-5 on one screen). Also when I resize the window, it takes 1-2 seconds until the change is done. This all seems to be connected to the redrawing. How did I "measure" the difference: -start both civlog version (with and without your patch) -load the same score-file in each (mine is 700 kb) -choose all players -now just start adding more data-types one by one from the top (Population,...). I notice a substantial speed decrease with your version vs. a not-so-bad speed-decrease in the original. When I come to pollution your version needs 3 seconds, the old one less than one sec to redraw. So the new one is about 3 times as slow. I know that you would ever not display that much info, it is only meant to make it easier to quantify the difference (as opposed to it "feeling" slow). > > While I was at it, I did a quick calc on mem usage of your old solution: > > Our hash (player, turn) -> (ScoreEntry) would take about: > > 20 players * 500 turns * 50 bytes = 500 KBytes of memory > > For every player and every turn you have a Hashtable > (ScoreEntry) containing for each data type two Integers (the key and the > value). > 20 player * 500 turns * (100 + 40*50) = 21 MByte > Hashtable ^ ^ Integers Huh? ScoreEntry has a Hash? OOps it is, but it should not. Of course ScoreEntry is not really part of the lookup-hash-structure. I admit it should not use a hash at all (it is a legacy from a time when information about data-types was not available). You can see that it can easily be changed to use a simple array with index = data_type. I have attached the modified file. I'd like to hear about it's memory consumption on your system (because it seems to differ from mine). On my system, it decreases the memory requirement of the superbig.log from the website from 36 MB to 26 MB. Now my above calc should be correct, yes? I only count the overhead for the hash lookup infrastructure, because the real data is now in arrays. But I'd like to point out that it is the runtime speed I am concerned about, not the memory. > > (the 50 bytes is an arbitrary number based on your info above, correct me > > if I am wrong). > > Oh it's normally 30 bytes I think. Then the overhead is only 300KB. > > Even if it's 100 bytes, this seems to be a very small price for *constant* > > access time to an arbitrary player/turn combination! > > > > If I understand your design correctly, you have at least a linear access > > time for each data-access (because you iterate the turns). > > I've done a binary search in the years array, so I had O(log N) access > time for one value and O(N*log N) in total. Ok. > > Since we also have to iterate the turns in Graph, we have a complexity of > > O(N) against O(N*N) with the new code. > > > > I just had another look at PlayerInfo and it looks very complicated. > > Some questions: > > - what is class IndexedList good for? It does the same as ArrayList. > > (IndexedList does not allow the insertion of Objets at an arbitrary > > positions either) > > > > But it does allow it. No, it just replaces the object (same as in ArrayList). Maybe I am just very confused here, but I don't see the difference to ArrayList. Please point it out to me in the code below. The method that does not exist in ArrayList is which one: /** * Allows the insertion of Objets at an arbitrary positions in * an array (not allowed in ArrayList). */ private class IndexedList { private YearValueMap[] content; IndexedList(int size) { content = new YearValueMap[size]; } void set(int idx, YearValueMap map) { if (idx >= content.length) grow(idx+1); content[idx] = map; } YearValueMap get(int idx) { return content[idx]; } private void grow(int newSize) { YearValueMap[] newMap = new YearValueMap[newSize]; System.arraycopy(content, 0, newMap, 0, content.length); content = newMap; } } > > I think this Array construction would only make sense if we do not save > > year->data but turn->data information. Then we could use turn as index > > on an array and have the combination of good access time (constant) > > and good memory usage. I know I am replying to myself here :), but I think we should really consider this, because it also makes the hole data-structure so much easier. Please let me know what you think about it. > > I also took a look at real mem usage (taskmgr). It was about 19 MB with your > > version, 23 MB with the 0.6.0 version. The difference seems to be negligible. > > > > With the superbig.log (2MB) it was even 50MB alone for the Score object. The absolute size seems to be a bit different on our systems. The superbig.log uses 36 MB on my system for the whole java-process. Considering it uses about 12 MB even when no file is loaded, this means the Score object is only 24 MB, so half the size of yours. Are you running on a 64 bit system? I can load a 3.3 MB file with the original version and only use 50 MB in total (java-process). BTW, the small attached change reduces it to 35 MB. > This has to do with the nature of java virtual machines and there memory > allocation (they claim memory fast and nearly never releases it): So for > reading all lines of the log file Score was storing thousands of String > objects. This String objects are realeased (and garbage collected) lateron Ahh, so the memory consumption comes from loading the score file and not only from the data-structure. If that is so, I would like to suggest that we keep the old data-structure and the new loadScoreFile-method. Would that be possible? Maybe it's only me, but I find the new data-structure hard to understand (not in principle but in the code), so I would vote for the simpler model. > but the jvm doesn't release it to other programs. > Nethertheless I have changed this. How? > Now the jvm uses 12MB after reading the > superbig.log. Thats very nice (although a little strange). It uses 12 MB *before* reading a file on my OS (W2k). I am definately looking forward to trying your new code with my bigger score-files. > > > > - the whole score thing should be using turns instead of years as primary > > > > reference. At the beginning everything used years than the GUI was > > > > changed to use turns and turn2year and year2turn was added. It > > > > would now be time to change score to also use turns. If we do a change to the data-structures, I would vote for changing it in that direction rather than all that complicated stuff now being introduced. It makes data-storage even easier: array indexed by turn (see above). BTW, who wrote that? [snip] > I hope I'm not too obtrusive. No, of course not. Actually, after writing the mail I was wondering if I could have formulated it more diplomatically. I really appreciate your sincere and objective answers. Michael -- eMail: mic...@gr... PGP key available on request |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 22:08:00
|
On Sun, Sep 24, 2000 at 10:28:03PM +0200, Ulrich Kuhn wrote: > Tomorrow I will send in all the modifications I'll mention here and > had earlier. > > On Sun, 24 Sep 2000, Michael Grundel wrote: > > I made an quick work-around by ignoring any player number "beyond" the > valid range. Maybe this is even a final solution because these additional > players have no name and how should one display them then? We should just generate a name like "Unnamed player 1". I don't think we have enough information to say that Player "Foobar" was splitted in turn 40 into "Foobar" and "Unnamned Player 2". > > > > - why do you use arrays in YearValueMap instead of a hash? Have you made > > > > some measurements? > > > > > > Not needed: > > > > Performance measurements are needed (it feels so slow). > > > > > http://www.javaworld.com/javaworld/jw-11-1999/jw-11-performance-2.html > > > > > > In short: (java) objects needs (besides the raw values) some memory to do > > > > Arrays.... now I understand why it is so slow :) > > Why do you think arrays are slow? I added some measurements to provide some hard facts we can discuss about. Summary: parsing can be improved. The data should be restructured like I mentioned in an earlier mail. > > To explain our existing data-structure: > > So far our efforts were never concerned with mem usage but with execution > > speed. > > > > To be honest, I think your implementation is too slow. I tried a relatively > > small logfile with only 5 players and even with that one it is too slow for > > my taste. > > I cannot test it with a big logfile because the bug prevents me from loading > > them. > > According to my testing (with top under Linux and java internal) "my > solution" is two times faster and uses the fifth part of memory than the > "old solution". I can confirm the two times faster. [ snip discussion about memory usage and data structures ] My opinion: change the data structes so they match nicely their usage in Graph. A lot of Ulrich's classes can be slimmed if standard classes are used. I don't think the memory is a problem as long as the VM is in core and not swapped. Raimar -- "Reality? That's where the pizza delivery guy comes from!" |
|
From: Ulrich K. <ku...@un...> - 2000-09-24 20:28:09
|
Tomorrow I will send in all the modifications I'll mention here and had earlier. On Sun, 24 Sep 2000, Michael Grundel wrote: > Hi! > > > > Please send comments. > > I really appreciate your contribution. But it seems to be very slow. > > There also is a bug that results in a > java.lang.ArrayIndexOutOfBoundsException > at org.freeciv.civlog.data.Score.loadScoreFile(Score.java:353) > at org.freeciv.civlog.gui.MainFrame.openFile(MainFrame.java:480) > > It affects almost all of my logfiles. It is the same one that is mentioned > by Deming. > > Ok, I just had a brief look at the code and I think your are making a wrong > assumption on the number of players. The number of players is not const! I made an quick work-around by ignoring any player number "beyond" the valid range. Maybe this is even a final solution because these additional players have no name and how should one display them then? > > > > - why do you use arrays in YearValueMap instead of a hash? Have you made > > > some measurements? > > > > Not needed: > > Performance measurements are needed (it feels so slow). > > > http://www.javaworld.com/javaworld/jw-11-1999/jw-11-performance-2.html > > > > In short: (java) objects needs (besides the raw values) some memory to do > > Arrays.... now I understand why it is so slow :) Why do you think arrays are slow? > > To explain our existing data-structure: > So far our efforts were never concerned with mem usage but with execution > speed. > > To be honest, I think your implementation is too slow. I tried a relatively > small logfile with only 5 players and even with that one it is too slow for > my taste. > I cannot test it with a big logfile because the bug prevents me from loading > them. According to my testing (with top under Linux and java internal) "my solution" is two times faster and uses the fifth part of memory than the "old solution". > > While I was at it, I did a quick calc on mem usage of your old solution: > Our hash (player, turn) -> (ScoreEntry) would take about: > 20 players * 500 turns * 50 bytes = 500 KBytes of memory For every player and every turn you have a Hashtable (ScoreEntry) containing for each data type two Integers (the key and the value). 20 player * 500 turns * (100 + 40*50) = 21 MByte Hashtable ^ ^ Integers > (the 50 bytes is an arbitrary number based on your info above, correct me > if I am wrong). Oh it's normally 30 bytes I think. > Even if it's 100 bytes, this seems to be a very small price for *constant* > access time to an arbitrary player/turn combination! > > If I understand your design correctly, you have at least a linear access > time for each data-access (because you iterate the turns). I've done a binary search in the years array, so I had O(log N) access time for one value and O(N*log N) in total. > Since we also have to iterate the turns in Graph, we have a complexity of > O(N) against O(N*N) with the new code. > > I just had another look at PlayerInfo and it looks very complicated. > Some questions: > - what is class IndexedList good for? It does the same as ArrayList. > (IndexedList does not allow the insertion of Objets at an arbitrary > positions either) > But it does allow it. > I think this Array construction would only make sense if we do not save > year->data but turn->data information. Then we could use turn as index > on an array and have the combination of good access time (constant) > and good memory usage. > > I also took a look at real mem usage (taskmgr). It was about 19 MB with your > version, 23 MB with the 0.6.0 version. The difference seems to be negligible. > With the superbig.log (2MB) it was even 50MB alone for the Score object. This has to do with the nature of java virtual machines and there memory allocation (they claim memory fast and nearly never releases it): So for reading all lines of the log file Score was storing thousands of String objects. This String objects are realeased (and garbage collected) lateron but the jvm doesn't release it to other programs. Nethertheless I have changed this. Now the jvm uses 12MB after reading the superbig.log. > > > - the whole score thing should be using turns instead of years as primary > > > reference. At the beginning everything used years than the GUI was > > > changed to use turns and turn2year and year2turn was added. It > > > would now be time to change score to also use turns. > > > > Yes. > > > > > - it would be nice to have an mapping (data type)->(boolean) which would > > > tell you which data types occur in the score file. Also a lastDataType > > > would be nice (can be used instead of > > > FreecivConstants.NUMBER_OF_DATA_TYPES to declare arrays) > > > > Question: Is the number and type of data types in a score file > > arbitrary? Especially can there be some types missing (0,1,4,8,11,...)? > > A comment in the freeciv-code suggests that new data-types are only added > at the end. So there are logfiles with different number of types but there > are no gaps (unless someone/someday takes out a type). > Would it be a problem to just ignore missing types? It is done so now. [the remaining part was nearly the same as above] I hope I'm not too obtrusive. Tomorrow I will send in all the modifications I mentioned here and earlier. Yours Ulrich |
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-24 16:59:01
|
> -----Original Message----- > From: civ...@li... > [mailto:civ...@li...]On Behalf Of Raimar > Falke > > Have you tried GIF or PNG? FYI. JPG use 'lossy' compression(therefore the noise around the text). Therefore jpg files is always smaller than GIF files which use 'loseless' compression(better quality). AFAIK PNG is in the GIF family? Oh! You mean GIF use 256 color therefore smaller. I tried it and it is much smaller. CivLog doesn't used much color therefore good with GIF. Will update the pics. tomorrow. -Deming -- I Love Linux |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 15:30:00
|
On Sun, Sep 24, 2000 at 10:41:27PM +0800, Wong TM (Huang Deming) wrote:
> </rubbish>*@!? Windoze, outlook cause page fault while editing....!@$@#$%
> $$$$$$sucker@#$#@$ </rubbish>
> > -----Original Message-----
> > From: civ...@li...
> > [mailto:civ...@li...]On Behalf Of Raimar
> > Falke
> >
> > On Sun, Sep 24, 2000 at 10:37:43AM +0800, Wong TM (Huang Deming) wrote:
> > >
> > [ AFAIK freeciv-dev would be a better place to such a proposal ]
> > It is also possible to omit lines if the value didn't change.
> > Or it may be
> > possible to compress the file. It looks like this is used for
> > the network
> > protocol and so zlib is a requirement. Nevertheless all these changes
> > complicate thing during the parsing. I don't know how many
> > programs are our
> > there which can parse the current format and will then need an update.
>
> At least quicker loading for smaller log.
I think the current format can be parsed straightforward. This is a big
advantage. We may look at our reading method and try to see what make it
slow (reading the file, parsing, creating internal data structures, setting
data in the internal data structures) It should be easy to comment some
parts out and measure the performance. I can also imagine a small C program
which parses the scorefile and produce a file in an interim format which can
be understand more easily be the java program.
> > > - Race info read along with player name.
> >
> > Where do you get the information for this?
>
> I haven't seen the other gamelog yet. I need to do this slowly. Will
> change report.c if needed. I haven't touch C for years but I sure I
> haven't lose the touch.
Nod.
Raimar
--
"With a PC, I always felt limited by the software available.
On Unix, I am limited by my knowledge."
-- Peter J. Schoenster <ps...@ba...>
|
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 15:21:39
|
On Sun, Sep 24, 2000 at 10:41:27PM +0800, Wong TM (Huang Deming) wrote: > > -----Original Message----- > > From: civ...@li... > > [mailto:civ...@li...]On Behalf Of Raimar > > Falke > > > > On Sat, Sep 23, 2000 at 09:41:51PM +0800, Wong TM (Huang Deming) wrote: > > > Done. I hope you mean the bigger screenshoot is ugly. > > > > You have updated them! Nevertheless you can still see > > artifacts around the > > text compare the attached small images. > > It just that I keep the *.jpg at around 100KB(1024*768) so that 56K modem > user can take a quick look at the shot. Should I use 800*600 but higher > quality. Have you tried GIF or PNG? Raimar -- pgp 2: id: 0F9D7955 len: 1024 fingerprint: 7760F933D5478009 4FA0C56F1DC2FB8E "Make it idiot-proof and someone will make a better idiot." |
|
From: Michael G. <mic...@gr...> - 2000-09-24 14:56:15
|
Hi!
> On Fri, 22 Sep 2000, Raimar Falke wrote:
>
> > On Thu, Sep 21, 2000 at 11:16:03PM +0200, Ulrich Kuhn wrote:
> > > Hi folks,
> > >
> > > today I have taken a closer look at the CivLog code and I stumbled across
> > > Score.java. It met my requirements: I understood it, I knew where to
> > > improve it, it was somewhat separated from the rest.
> > > So I wrote a in my opinion improved version of it.
> > > I 've attached the three necessary files in a zip file. PlayerInfo.java
> > > and Searcher.java are new files.
> > >
> > > Please send comments.
I really appreciate your contribution. But it seems to be very slow.
There also is a bug that results in a
java.lang.ArrayIndexOutOfBoundsException
at org.freeciv.civlog.data.Score.loadScoreFile(Score.java:353)
at org.freeciv.civlog.gui.MainFrame.openFile(MainFrame.java:480)
It affects almost all of my logfiles. It is the same one that is mentioned
by Deming.
Ok, I just had a brief look at the code and I think your are making a wrong
assumption on the number of players. The number of players is not const!
You calc the max munber of players based on the player-list at the beginning
of the logfile. This does not take into account that during the game more
players can be spawned (and additional log-entries are generated).
> > - why do you use arrays in YearValueMap instead of a hash? Have you made
> > some measurements?
>
> Not needed:
Performance measurements are needed (it feels so slow).
> http://www.javaworld.com/javaworld/jw-11-1999/jw-11-performance-2.html
>
> In short: (java) objects needs (besides the raw values) some memory to do
> things like synchronization. This extra memory varies it's between 10-40
> bytes per object. So the object (class IntW { int i; }) will not use 4
> bytes but perhaps 20.
> The idea to store the values in a hash was not bad but for pure value
> storage objects are using way to much memory. BTW: arrays are objects too
> but an array containing many values is only one object...
Arrays.... now I understand why it is so slow :)
To explain our existing data-structure:
So far our efforts were never concerned with mem usage but with execution
speed.
To be honest, I think your implementation is too slow. I tried a relatively
small logfile with only 5 players and even with that one it is too slow for
my taste.
I cannot test it with a big logfile because the bug prevents me from loading
them.
While I was at it, I did a quick calc on mem usage of your old solution:
Our hash (player, turn) -> (ScoreEntry) would take about:
20 players * 500 turns * 50 bytes = 500 KBytes of memory
(the 50 bytes is an arbitrary number based on your info above, correct me
if I am wrong).
Even if it's 100 bytes, this seems to be a very small price for *constant*
access time to an arbitrary player/turn combination!
If I understand your design correctly, you have at least a linear access
time for each data-access (because you iterate the turns).
Since we also have to iterate the turns in Graph, we have a complexity of
O(N) against O(N*N) with the new code.
I just had another look at PlayerInfo and it looks very complicated.
Some questions:
- what is class IndexedList good for? It does the same as ArrayList.
(IndexedList does not allow the insertion of Objets at an arbitrary
positions either)
I think this Array construction would only make sense if we do not save
year->data but turn->data information. Then we could use turn as index
on an array and have the combination of good access time (constant)
and good memory usage.
I also took a look at real mem usage (taskmgr). It was about 19 MB with your
version, 23 MB with the 0.6.0 version. The difference seems to be negligible.
> > - the whole score thing should be using turns instead of years as primary
> > reference. At the beginning everything used years than the GUI was
> > changed to use turns and turn2year and year2turn was added. It
> > would now be time to change score to also use turns.
>
> Yes.
>
> > - it would be nice to have an mapping (data type)->(boolean) which would
> > tell you which data types occur in the score file. Also a lastDataType
> > would be nice (can be used instead of
> > FreecivConstants.NUMBER_OF_DATA_TYPES to declare arrays)
>
> Question: Is the number and type of data types in a score file
> arbitrary? Especially can there be some types missing (0,1,4,8,11,...)?
A comment in the freeciv-code suggests that new data-types are only added
at the end. So there are logfiles with different number of types but there
are no gaps (unless someone/someday takes out a type).
Would it be a problem to just ignore missing types?
> > - in
> > PlayerInfo(String name)
> > {
> > playerName = name;
> > typeList = new IndexedList(28);
> > }
> > Can you replace the 28 with a more meaningful number?
>
> IndexedList is able to grow if needed. So 28 is just an initial value (I
> think there are 27 or 28 data types but maybe there are (later) more.
As I pointed out above, I don't like the current implemtation of the lists for
performance reasons.
But if you need number of data types, use FreecivConstants.NUMBER_OF_DATA_TYPES.
> > - lets try to describe the data structure:
> > * Score has PlayerInfoS which are indexed by player number
> > * PlayerInfo (holds all data about one player) has IndexedList
> > * IndexedList (holds all data about one player) has YearValueMapS which
> > are indexed by data type
> > * YearValueMap has pairs of (year, value) which are indexed by year
>
> Seems to be correct but maybe this is easier to understand:
> * Score contains PlayerInfoS indexed by player number.
> * PlayerInfo contains a IndexedList indexed by data type.
> * Each entry in this IndexedList is a YearValueMap (one for every data
> type) that contains all values for one data type (and player).
> Well, not quite easier but at least another formulation. :)
Thanks for the description.
> > Current structures:
> > * Score has pairs (EntryKey,ScoreEntry) which are indexed by EntryKey
> > * EntryKey describes all data of one player at one year
> > * ScoreEntry (holds all data of one player at one year) has pairs of
> > (type, value) which are indexed by type
> > So you don't use EntryKey. You added some layers to the prior flat
> > structure.
>
> Yes one layer more.
And some loops because of the arrays. :)
Michael
--
eMail: gr...@pi...
PGP key available on request
|
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-24 14:40:31
|
</rubbish>*@!? Windoze, outlook cause page fault while editing....!@$@#$% $$$$$$sucker@#$#@$ </rubbish> > -----Original Message----- > From: civ...@li... > [mailto:civ...@li...]On Behalf Of Raimar > Falke > > On Sun, Sep 24, 2000 at 10:37:43AM +0800, Wong TM (Huang Deming) wrote: > > > [ AFAIK freeciv-dev would be a better place to such a proposal ] > It is also possible to omit lines if the value didn't change. > Or it may be > possible to compress the file. It looks like this is used for > the network > protocol and so zlib is a requirement. Nevertheless all these changes > complicate thing during the parsing. I don't know how many > programs are our > there which can parse the current format and will then need an update. At least quicker loading for smaller log. zlib is alien to me right now(I know what, I don't know how). > > - Race info read along with player name. > > Where do you get the information for this? I haven't seen the other gamelog yet. I need to do this slowly. Will change report.c if needed. I haven't touch C for years but I sure I haven't lose the touch. -Deming "Please remain calm...I may be mad, but I am a professional." --Mad Scientist |
|
From: Wong TM (H. Deming) <loo...@ma...> - 2000-09-24 14:40:28
|
> -----Original Message----- > From: civ...@li... > [mailto:civ...@li...]On Behalf Of Raimar > Falke > > On Sat, Sep 23, 2000 at 09:41:51PM +0800, Wong TM (Huang Deming) wrote: > > Done. I hope you mean the bigger screenshoot is ugly. > > You have updated them! Nevertheless you can still see > artifacts around the > text compare the attached small images. It just that I keep the *.jpg at around 100KB(1024*768) so that 56K modem user can take a quick look at the shot. Should I use 800*600 but higher quality. > -- > "Despite all the medical advances of the 20th century, the mortality > rate remains unchanged at 1 death per person." a new sig. :) - Deming "Please remain calm...I may be mad, but I am a professional." --Mad Scientist |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 12:20:26
|
On Sun, Sep 24, 2000 at 10:37:43AM +0800, Wong TM (Huang Deming) wrote: > > > I see that the log file can get very big. > > Idea: > > - stop outputing dead player log and maybe add a 'dead' tag to signify > end of dead player log. [ AFAIK freeciv-dev would be a better place to such a proposal ] It is also possible to omit lines if the value didn't change. Or it may be possible to compress the file. It looks like this is used for the network protocol and so zlib is a requirement. Nevertheless all these changes complicate thing during the parsing. I don't know how many programs are our there which can parse the current format and will then need an update. > - Race info read along with player name. Where do you get the information for this? Raimar -- "I haven't lost my mind - it's backed up on tape somewhere." |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 12:14:05
|
On Sat, Sep 23, 2000 at 09:41:51PM +0800, Wong TM (Huang Deming) wrote: > Done. I hope you mean the bigger screenshoot is ugly. You have updated them! Nevertheless you can still see artifacts around the text compare the attached small images. > btw. Any free webhosting site you will recommend as the current asia1 > doesn't support some advance ftp command(it tedious to update the site > manually). Sorry I can't help you on this. For CivLog you may look at the web hosting service sourceforge offers. Otherwise you can write scripts which do the uploading. Raimar -- "Despite all the medical advances of the 20th century, the mortality rate remains unchanged at 1 death per person." |
|
From: Raimar F. <hawk@A315-2b.WH8.TU-Dresden.De> - 2000-09-24 11:54:32
|
On Sat, Sep 23, 2000 at 11:36:42AM +0200, Ulrich Kuhn wrote: > On Fri, 22 Sep 2000, Raimar Falke wrote: > > > On Thu, Sep 21, 2000 at 11:16:03PM +0200, Ulrich Kuhn wrote: > > > Hi folks, > > > > > > today I have taken a closer look at the CivLog code and I stumbled across > > > Score.java. It met my requirements: I understood it, I knew where to > > > improve it, it was somewhat separated from the rest. > > > So I wrote a in my opinion improved version of it. > > > > > > I 've attached the three necessary files in a zip file. PlayerInfo.java > > > and Searcher.java are new files. > > > > > > Please send comments. > > > > It would be nice if you had sent a short diescription how the new score > > works and what kind of improvement you made. I took a look at the code > > and have to following comments: > > - can you go over the method declaritions and set the permission (private, > > protected or public)? Some methods in PlayerInfo for example should be > > private or public but not package > > I think most of the method declarations are useful: > PlayerInfo is ment to be a helper and data storage for Score (methods > can't be private) but PlayerInfo is not intended to be given away by > Score (so no public methods). Sounds good. If Score doesn't leak any reference about PlayerInfo outside you can also declare the methods public. > But I need an value access/exchange object for Score that can other > objects (ie Graph) request. So maybe I'll change this... A replacement for ScoreEntry? > > - why do you use arrays in YearValueMap instead of a hash? Have you made > > some measurements? > > Not needed: > http://www.javaworld.com/javaworld/jw-11-1999/jw-11-performance-2.html > > In short: (java) objects needs (besides the raw values) some memory to do > things like synchronization. This extra memory varies it's between 10-40 > bytes per object. So the object (class IntW { int i; }) will not use 4 > bytes but perhaps 20. > The idea to store the values in a hash was not bad but for pure value > storage objects are using way to much memory. BTW: arrays are objects too > but an array containing many values is only one object... This is something I would expect. However there is a tradeoff between performance and code size/maintainability. YearValueMap is around 90 lines + 50 lines from Searcher vs 30 lines of ScoreEntry (it isn't a valid comparison because of the comments). So we can more code it we didn't reuse the classes java provides. So the define the break even point of this we have to specify what performance on what target computer we expect. We also have to measure the oerformance. It is a harder to measure the maintainability. > > - it would be nice to have an mapping (data type)->(boolean) which would > > tell you which data types occur in the score file. Also a lastDataType > > would be nice (can be used instead of > > FreecivConstants.NUMBER_OF_DATA_TYPES to declare arrays) > > Question: Is the number and type of data types in a score file > arbitrary? Especially can there be some types missing (0,1,4,8,11,...)? Not very likely. > > - the semantics of findEntry are wrong. findEntry in the current version > > returns null if for this year no entry is found. We can get rid of this > > kludge if we add an isValidYear() or maybe this also goes away if we > > change everything to use turns. > > Is fixed but I'll remove this method anyway because it uses ScoreEntry. Since we now have users of Score we can optimize it in this respect: from Graph: for (int x=startX; x<endX; x++) { int year; if(view.isDisplayLabeledInYears()) year=x; else year=score.turn2year(x); /** invalid year, skip it */ if (score.findEntry (year, 0) == null) continue; for (int index=0; index<indexToSelectedPlayer.length;index++) { int player=indexToSelectedPlayer[index]; ScoreEntry current = score.findEntry (year, player); float y=current.getData (myDataType); if(perCity && FreecivConstants.DATA_PER_CITY[myDataType]) { int cities=current.getData (FreecivConstants.CITIES); if(cities>0) y/=(float)cities; } PolyLine theLine = lines.getLine(index); theLine.addPoint ((float)x, y); }/* end - inner for (index) */ }/* end - outer for (year) */ If we change the outer loop to use turns, we would need something like: SomeResultClass getInfoForOneTurnAndOneDataType(int turn, int dataType) And SomeResultClass has a method like: int getValueForPlayer(int player) > > - lets try to describe the data structure: > > * Score has PlayerInfoS which are indexed by player number > > * PlayerInfo (holds all data about one player) has IndexedList > > * IndexedList (holds all data about one player) has YearValueMapS which > > are indexed by data type > > * YearValueMap has pairs of (year, value) which are indexed by year > > Seems to be correct but maybe this is easier to understand: > * Score contains PlayerInfoS indexed by player number. > * PlayerInfo contains a IndexedList indexed by data type. > * Each entry in this IndexedList is a YearValueMap (one for every data > type) that contains all values for one data type (and player). > Well, not quite easier but at least another formulation. :) > > > Current structures: > > * Score has pairs (EntryKey,ScoreEntry) which are indexed by EntryKey > > * EntryKey describes all data of one player at one year > > * ScoreEntry (holds all data of one player at one year) has pairs of > > (type, value) which are indexed by type > > So you don't use EntryKey. You added some layers to the prior flat > > structure. > > Yes one layer more. I'm still not convinced what advantages we get can from this extra layer. The code size increases and the proposed structure above (SomeResultClass) will likely get better performance measurements. Raimar -- Tank: So what do you need? Besides a miracle. Neo: Guns. Lots of guns. -- From The Matrix |