From: Johannes G. <joh...@gm...> - 2007-06-06 22:19:50
Attachments:
mmap.patch
|
Dear developers, I have made a test implementation for star catalog loading using mmap. I am really astonished, because now, with only 1GB of ram on my desktop, I can easily load up to stars_8, without any loading delay, and without any noticable performance decrease during operation. Well, the explanation is, that most of the stars are never displayed. Matthew, this patch is for you and your tiny computer with only 512MB. If you dare, please try stars_8 with this patch. I would be interested if you notice any shortage of memory during long and intensive program use, or if the operating system releases unneeded pages. The patch is against current SVN (which is already different from 0.9). It crashes upon program termination, but anyway it's just an experiment. Johannes |
From: Johannes G. <joh...@gm...> - 2007-06-06 22:43:25
Attachments:
mmap.patch
|
> > The patch is against current SVN (which is already different from 0.9). > It crashes upon program termination, but anyway it's just an experiment. I have created a better patch that does not crash on exit. Johannes |
From: Matthew G. <mat...@gm...> - 2007-06-06 23:31:09
|
I am now downloading the largest file, which I did not previous have a local copy of. With files 0 - 7 it works well. The load time at the start of the program is all but eliinated. I notice some non-smooth operation when zooming into dense regions of the milky way, but this soon calms down. I ran "ps u -p $STEL_PID" every second during this procedure. When the non-smooth operation was happening was when there was disk activity, and at this point the real memory usage increased. It did not decrease at any point. The VSS value was constant. I think it is probably preferable for many users, but we should keep the option to load the old way too. I expect longer load times are less of a problem for planetarium users, who want the smoothest possible operation once loading is complete. As I reach this point in the message, the last file is about to finish downloading... [some moments later] Yes, it works very well, even with file 8 too! It is wonderful to see so many stars the zoom is increased, really wonderful! It a little jerky, but really not so bad. |
From: <fab...@go...> - 2007-06-12 11:55:05
|
Hi, Well done for the mmap patch. If we want to include that in the SVN, we should make sure that it is portable to other plateforms, and use a fallback if needed. There should be code in the CMakeLists.txt which check for mmap and define HAVE_MMAP in the config.h or something like that. Fabien On 6/7/07, Matthew Gates <mat...@gm...> wrote: > > I am now downloading the largest file, which I did not previous have a > local > copy of. > > With files 0 - 7 it works well. The load time at the start of the program > is all but eliinated. I notice some non-smooth operation when zooming > into > dense regions of the milky way, but this soon calms down. > > I ran "ps u -p $STEL_PID" every second during this procedure. When the > non-smooth operation was happening was when there was disk activity, and > at > this point the real memory usage increased. It did not decrease at any > point. The VSS value was constant. > > I think it is probably preferable for many users, but we should keep the > option to load the old way too. > > I expect longer load times are less of a problem for planetarium users, > who > want the smoothest possible operation once loading is complete. > > As I reach this point in the message, the last file is about to finish > downloading... > > [some moments later] > > Yes, it works very well, even with file 8 too! It is wonderful to see so > many stars the zoom is increased, really wonderful! It a little jerky, > but > really not so bad. > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Stellarium-pubdevel mailing list > Ste...@li... > https://lists.sourceforge.net/lists/listinfo/stellarium-pubdevel > |
From: Johannes G. <joh...@gm...> - 2007-06-12 20:10:14
|
On 2007.06.12 13:54:55 CEST, Fabien Chéreau wrote: > Hi, > > Well done for the mmap patch. If we want to include that in the SVN, we > should make sure that it is portable to other plateforms, and use a > fallback if needed. > There should be code in the CMakeLists.txt which check for mmap and define > HAVE_MMAP in the config.h or something like that. We don't need HAVE_MMAP, because mmap exists everywhere. Even in Windows95, although it is called differently. There are other problems: 1) smooth operation vs. having the data on disk. Some people will prefer preloading of all data because this guarantees smooth operation without disk access (provided you have enough RAM). 2) binary compatibility. The current patch work only for little endian machines, and I am unsure about compilers other than gcc. Well, this is not such a restriction: only the old Motorola G4 Macs and Sun, Hp, SGI, Alpha machines are out of luck. Conversion to native binary format can be done A) before starting of stellarium B) right after loading the data in stellarium C) before rendering of each single star. Currently B is implemented, but this is not possible with mmap. Maybe we will have to do both A and C. In any case it would be good to see how big the performance drawback is of method C. Johannes |
From: Johannes G. <joh...@gm...> - 2007-06-27 21:41:50
|
On 2007.06.12 22:09:59 CEST, Johannes Gajdosik wrote: > On 2007.06.12 13:54:55 CEST, Fabien Chéreau wrote: > > Hi, > > > > Well done for the mmap patch. If we want to include that in the SVN, we > > should make sure that it is portable to other plateforms, and use a > > fallback if needed. > > There should be code in the CMakeLists.txt which check for mmap and define > > HAVE_MMAP in the config.h or something like that. > > We don't need HAVE_MMAP, because mmap exists everywhere. Even in Windows95, > although it is called differently. > > There are other problems: > > 1) smooth operation vs. having the data on disk. > Some people will prefer preloading of all data because this guarantees smooth operation > without disk access (provided you have enough RAM). > > 2) binary compatibility. > The current patch work only for little endian machines, and I am unsure about compilers > other than gcc. Well, this is not such a restriction: only the old Motorola G4 Macs > and Sun, Hp, SGI, Alpha machines are out of luck. Conversion to native binary format can be done > > A) before starting of stellarium > B) right after loading the data in stellarium > C) before rendering of each single star. > > Currently B is implemented, but this is not possible with mmap. > Maybe we will have to do both A and C. In any case it would be good to see how big > the performance drawback is of method C. > > Johannes I have committed a first version of the mmap patch. The restriction is that it currently works for little endian machines only, so users of G4-Macs are out of luck for now. Nigel, If you want something for your old Mac, I can do it for you. You just have to compile and test. In order to use the mmap feature you must write in stars.ini: cat_file_name_05 = mmap:stars_5_1v0_0.cat cat_file_name_06 = mmap:stars_6_2v0_0.cat cat_file_name_07 = mmap:stars_7_2v0_0.cat cat_file_name_08 = mmap:stars_8_2v0_01.cat So mmapping can be enabled/disabled for single files. The fallback is the current loading into main memory. I have implemented a windows version too, but was not even able to compile it, because I currently have no working crosscompiling environment. Yours, Johannes |