From: <ben...@ug...> - 2007-01-15 21:05:11
|
Hi, first let me tell I'm European, non-english speaking, so most of your abbreviations are like chinese to me. ;-) And unfortunately, I wouldn't know where to put Virginia on a map, apart from it being in the US... > And I just burned off a copy of FC6 Live. But I have NDI if either has > gramps on the cd. She does not have a net connection, so that's going to > cramp things a bit. We did discuss it, but I'm not sure if she is > educated just yet about its advantages and disadvantages. The ideal > situation would be for her to get hooked up, and setup a vpn from her box > to the historical societies box, but I have NDI if they have connectivity > there either. Dialup due to the distance to the CO most certainly if > they do. you would be surprised what genealogist are able to do with PC's. Most have internet connectivity as it saves the drive to some archive in many cases. I forgot to mention, there is a live cd available: http://gramps-project.org/index.php?module=pagemaster&PAGE_user_op=view_page&PAGE_id=19&MMN_position=33:9 It has several open source genealogy tools on it, and is based on Ubuntu Dapper Drake LTS, which is a good choice for a home user not living on the edge! However, then you best upgrade to the latest ubuntu package from here: http://sourceforge.net/project/showfiles.php?group_id=25770 after install. > I'll have to check and see what format their existing systems are using as > it would be a huge plus to just be able to read in their book of cd's and > go from there. If they are doing genealogy, they will have some data in GEDCOM. You can import and export to GEDCOM from gramps. Best is to make a new empty grdb database, and import a GEDCOM into it. Some info is in text or spreadsheets. A third party plugin (see bottom of http://developers.gramps-project.org/tiki-index.php?page=ToolSpecifications , CSV import) might be handy there. Note that problems with that should be send to the author, not gramps. >> The database should scale well, the other two not (as the file must be >> read in completely, indexes build, references made, ...). > > Uhgg. OTOH, this ain't googles engine we're building here folks, and > speed may not be a prerequisite until such time as there is enough data > to make it usefull. Then, throwing money at it would appear to be at > least a partial solution. This box is only a gigahertz cpu, max, maybe > slower. A gigahertz should be just fine. >> Of note here: the .gramps format is the smallest, whereas the .grdb >> format is a >> database and contains indexes, .. and hence is MUCH larger than the >> other formats (you gain speed with this off course). > > Sounds rather like the way to go as long as it could still export in a > format that would constitute an interchange to the GEDCOM world if that's > what's being used now. Yes, perfectly possible. Gramps can contain more info than GEDCOM, but that should not be a problem. Do set up a backup mechanism however if you feel up to it. Making a db is a lot of work, and new users often do strange things software was not meant to do. Also, GRAMPS has a good manual (separate package on ubuntu), make your friend read it, or read it yourself to assist her. It is not a long manual. [snip] > Another consideration is that the majority of this group can be cataloged > as being in the over 60 area, and computer literacy if they ever had any > is in its declining years. In that regard I'm no different. I wrote > good code, in assembly 15-20 years ago, and I still grok that, but even a > bash script on modern machinery is a struggle for me at 72. So we may be > biting off more than we can chew here, but it is worth it just to try > IMO. Linux is for children these days ;-) It's not as it used to be. The older people I see in our city archive are very good at using the computers for their purposes. I am amazed sometimes. > I find it interesting because I'm the "furriner", only lived here 22 years > now, coming to West Virginia in 1984 to be the CE at WDTV & from which > I've been trying to retire from since 2001 but they won't let me. > > Very interesting because I have some of the major battlefields that made > this country what it once was, _free_, are just an hour or three down the > road. Interesting places to visit since prior to that, I had only read > about them in the history books in school back in the '40's. If you go > there now and listen closely, the echo's of the woundeds screams can > still be heard, particularly if you have a good imagination. :) One of > them is where 50,000 men met their maker in one day back then. And you > really do feel the history when walking around in such a place. The tree > of liberty was truly refreshed with the blood of patriots that day. Now > its dying of malnutrition. I know what you mean. I grew up 4 km of Passendale. Many people don't know the battle: WO I, the British wanted to beat the Germans before the US arrived... From july to november, less than 6 miles of terrain was won. See http://en.wikipedia.org/wiki/Passendale (Note, they say they lost 570,000 troops, however the wiki is inaccurate in that lost meant in those days not being able to fight anymore, some 30% probably died, no-one really knows.) Chilling idea, to play on those fields, even stranger that all of the crosses bear names from abroad... B. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
From: Gene H. <gen...@ve...> - 2007-01-14 23:34:44
|
On Sunday 14 January 2007 17:34, ben...@ug... wrote: >Hi, > >first let me tell I'm European, non-english speaking, so most of your >abbreviations are like chinese to me. ;-) And unfortunately, I wouldn't > know where to put Virginia on a map, apart from it being in the US... East Central, below Pennsylvania, and west of Maryland and Virginia, both of those have salt water eastern shores. And your english is at least as good as mine. >> And I just burned off a copy of FC6 Live. But I have NDI =No Darned Idea (or something long those lines :) >> if either >> has gramps on the cd. She does not have a net connection, so that's >> going to cramp things a bit. We did discuss it, but I'm not sure if >> she is educated just yet about its advantages and disadvantages. The >> ideal situation would be for her to get hooked up, and setup a vpn >> from her box to the historical societies box, but I have NDI if they >> have connectivity there either. Dialup due to the distance to the CO =Central Office (of the local telco) >> most certainly if they do. > >you would be surprised what genealogist are able to do with PC's. Most > have internet connectivity as it saves the drive to some archive in > many cases. I forgot to mention, there is a live cd available: >http://gramps-project.org/index.php?module=pagemaster&PAGE_user_op=view_ >page&PAGE_id=19&MMN_position=33:9 It has several open source genealogy > tools on it, and is based on Ubuntu Dapper >Drake LTS, which is a good choice for a home user not living on the > edge! However, then you best upgrade to the latest ubuntu package from > here: http://sourceforge.net/project/showfiles.php?group_id=25770 after > install. 1.0 says Live, 2.0 says Desktop, I assume its not a 'Live' image then? I'm pulling 1.0 as I type. >> I'll have to check and see what format their existing systems are >> using as it would be a huge plus to just be able to read in their book >> of cd's and go from there. > >If they are doing genealogy, they will have some data in GEDCOM. You >can import >and export to GEDCOM from gramps. Best is to make a new empty grdb > database, and import a GEDCOM into it. >Some info is in text or spreadsheets. A third party plugin (see bottom > of > http://developers.gramps-project.org/tiki-index.php?page=ToolSpecificat >ions , CSV import) might be handy there. Note that problems with that > should be send to the author, not gramps. > >>> The database should scale well, the other two not (as the file must >>> be read in completely, indexes build, references made, ...). >> >> Uhgg. OTOH, =On The Other Hand >> this ain't googles engine we're building here folks, and >> speed may not be a prerequisite until such time as there is enough >> data to make it usefull. Then, throwing money at it would appear to >> be at least a partial solution. This box is only a gigahertz cpu, >> max, maybe slower. > >A gigahertz should be just fine. Yes, but its 10GB drive is a very small teacup, easily over filled. >>> Of note here: the .gramps format is the smallest, whereas the .grdb >>> format is a >>> database and contains indexes, .. and hence is MUCH larger than the >>> other formats (you gain speed with this off course). >> >> Sounds rather like the way to go as long as it could still export in a >> format that would constitute an interchange to the GEDCOM world if >> that's what's being used now. > >Yes, perfectly possible. Gramps can contain more info than GEDCOM, but > that should not be a problem. >Do set up a backup mechanism however if you feel up to it. Making a db >is a lot >of work, and new users often do strange things software was not meant to > do. Also, GRAMPS has a good manual (separate package on ubuntu), make > your friend read it, or read it yourself to assist her. It is not a > long manual. I'll get it, thanks. >[snip] > >> Another consideration is that the majority of this group can be >> cataloged as being in the over 60 area, and computer literacy if they >> ever had any is in its declining years. In that regard I'm no >> different. I wrote good code, in assembly 15-20 years ago, and I >> still grok that, but even a bash script on modern machinery is a >> struggle for me at 72. So we may be biting off more than we can chew >> here, but it is worth it just to try IMO. =In My Opinion >Linux is for children these days ;-) It's not as it used to be. >The older people I see in our city archive are very good at using the >computers >for their purposes. I am amazed sometimes. Just don't change things on them once they get used to it. :) >> I find it interesting because I'm the "furriner", only lived here 22 >> years now, coming to West Virginia in 1984 to be the CE =Chief Engineer, aka their Mr. Fixit. I am a Certified Electronics Technician and a general Jack of all trades, aka a JOAT. >> at WDTV a tv station, the CBS network affiliate for north central West Virginia, and note that West Virginia is a separate state from Virginia. >> & from >> which I've been trying to retire from since 2001 but they won't let >> me. >> >> Very interesting because I have some of the major battlefields that >> made this country what it once was, _free_, are just an hour or three >> down the road. Interesting places to visit since prior to that, I had >> only read about them in the history books in school back in the '40's. >> If you go there now and listen closely, the echo's of the woundeds >> screams can still be heard, particularly if you have a good >> imagination. :) One of them is where 50,000 men met their maker in >> one day back then. And you really do feel the history when walking >> around in such a place. The tree of liberty was truly refreshed with >> the blood of patriots that day. Now its dying of malnutrition. > >I know what you mean. I grew up 4 km of Passendale. Many people don't > know the battle: WO I, the British wanted to beat the Germans before > the US arrived... From july to november, less than 6 miles of terrain > was won. See http://en.wikipedia.org/wiki/Passendale (Note, they say > they lost 570,000 troops, however the wiki is inaccurate in that lost > meant in those days not being able to fight anymore, some 30% probably > died, no-one really knows.) Chilling idea, to play on those fields, > even stranger that all of the crosses bear names from abroad... No doubt, and some of the later ones were no doubt West Virginia's famous Mountaineers. But the death percentage figures in those days was probably higher than 30% due to the poorer medical care and sanitation. What is a flesh wound today was fatal more often than not back then. Infections. Lots of staph. Our hills are fair sized ones here, local, right up in your face real estate. One local place I've hunted deer on in the past, the owner used to claim he could farm both sides of a few of his acres, but told me not to tell the tax assessor. :-) I can gain 100 meters within 300 meters of my house. Any flat land you want, you have to make it flat somehow after you buy it. Having climbed that place back when my legs were a bit more able, I can't argue with him, some of it was darned near straight up. But, I did bring a nice buck off those hills the last time I was there. Plumb tuckered by the time I got to my truck with it though, I had to drag him across 2 drainages to get there. Thanks to West Virginia's tendency toward smaller deer, it was doable. Thanks Benny. -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2007 by Maurice Eugene Heskett, all rights reserved. |
From: Alex R. <sh...@gr...> - 2007-01-15 05:48:17
|
Gene, On Sun, 2007-01-14 at 18:34 -0500, Gene Heskett wrote: > 1.0 says Live, 2.0 says Desktop, I assume its not a 'Live' image then? =20 > I'm pulling 1.0 as I type. Desktop has both Live session on it and the way to install if the user wishes. It is all explained on that page. Don't use 1.0, use 2.0. Alex --=20 Alexander Roitman http://www.gramps-project.org |
From: Gene H. <gen...@ve...> - 2007-01-15 11:00:40
|
On Monday 15 January 2007 00:45, Alex Roitman wrote: >Gene, > >On Sun, 2007-01-14 at 18:34 -0500, Gene Heskett wrote: >> 1.0 says Live, 2.0 says Desktop, I assume its not a 'Live' image then? >> I'm pulling 1.0 as I type. > >Desktop has both Live session on it and the way to install if the user >wishes. It is all explained on that page. Don't use 1.0, use 2.0. > >Alex I didn't go in on html, but ftp. So I didn't see that advisory, my bad. Thanks Alex, I just got a verse of the charge out of k3b, so its ready to go. Now all I have to do is find the time, which is going to be in somewhat short supply for a couple of weeks or more while I rebuild a tv station's transmitter. I hope the wholesalers are open tomorrow, as its Martin Luther King Day, today already according to my clock. -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2007 by Maurice Eugene Heskett, all rights reserved. |
From: Douglas S. B. <db...@cs...> - 2007-01-15 16:06:58
|
Some data on the topic of big GRAMPS databases: I recently received a GEDCOM from a relative that has over 100,000 people in it. It was about 5 MB compressed zip GEDCOM. I uncompressed it, to about 20 MB, I think. I'm running gramps 2.2.3. I imported it into GRAMPS, and that took awhile---I let it run when I went to bed. Now, when I open the file, it takes a good 5 minutes, and some functions are too slow to be useful (running on a 4 year old computer, couple of GHz). The GRDB is about 400 MB. But it is functional, if I am patient. I think that the speed issues can be improved. GRAMPS has gone through a big transformation in the last year, and I suspect that there are speed-ups to be found in many places. BTW, Python is pretty easy to pick up, and you should join the gramps developers mailing list if interested. What is the largest GRAMPS database that you all have worked on? -Doug |
From: Kees B. <kee...@xs...> - 2007-01-15 19:44:34
|
Op maandag 15 januari 2007 17:06, schreef Douglas S. Blank: > Some data on the topic of big GRAMPS databases: I recently received a > GEDCOM from a relative that has over 100,000 people in it. It was about 5 > MB compressed zip GEDCOM. I uncompressed it, to about 20 MB, I think. > > I'm running gramps 2.2.3. I imported it into GRAMPS, and that took > awhile---I let it run when I went to bed. Now, when I open the file, it > takes a good 5 minutes, and some functions are too slow to be useful > (running on a 4 year old computer, couple of GHz). The GRDB is about 400 > MB. But it is functional, if I am patient. > > I think that the speed issues can be improved. GRAMPS has gone through a > big transformation in the last year, and I suspect that there are > speed-ups to be found in many places. BTW, Python is pretty easy to pick > up, and you should join the gramps developers mailing list if interested. > > What is the largest GRAMPS database that you all have worked on? Here are some figures about my father's database (on an AMD 3200+ with 1Gb mem). - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes about 3,5 minutes - the resulting database is about 100Mb - opening that 100Mb database takes roughly 30-40 seconds - in people view the clicking on the little triangles is very confusing because of the slow feedback (When I showed this to my 84 year old father he clicked a few times on a triangle and nothing happened. It took 5 seconds before the list unfolded. That is, a surname list with a lot of people, which is not unusual if you have a genealogy for your own family. Now if you click twice it takes 10 seconds without any visible result.) - scrolling in people view using PgUp and PgDn will confuse the display of the cursor line (the blue line with the selected person) The last two items make Gramps simply not-usable for my father. BTW. This is by no means a complaint, but it is something that we must be aware of now that Gramps is in a state that more people are willing to try it out. -- Kees |
From: Richard T. <rjt...@th...> - 2007-01-15 20:10:30
|
On Monday 15 January 2007 19:44, Kees Bakker wrote: > > Here are some figures about my father's database (on an AMD 3200+ with 1Gb > mem). > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes about > 3,5 minutes > - the resulting database is about 100Mb > - opening that 100Mb database takes roughly 30-40 seconds > - in people view the clicking on the little triangles is very confusing > because of the slow feedback (When I showed this to my 84 year old father > he clicked a few times on a triangle and nothing happened. It took 5 > seconds before the list unfolded. That is, a surname list with a lot of > people, which is not unusual if you have a genealogy for your own family. > Now if you click twice it takes 10 seconds without any visible result.) > - scrolling in people view using PgUp and PgDn will confuse the display of > the cursor line (the blue line with the selected person) > > The last two items make Gramps simply not-usable for my father. > Would you say that it is the speed (or lack of) that is the primary problem or the lack of feedback? It may be possible to add some user feedback for operations that are taking a long time. I am not sure that it will ever be possible to make a database with 100,000+ records work very quickly in gramps. The View model will always be slow with that many records. Richard |
From: Kees B. <kee...@xs...> - 2007-01-15 21:12:45
|
Op maandag 15 januari 2007 21:13, schreef Richard Taylor: > On Monday 15 January 2007 19:44, Kees Bakker wrote: > > > > Here are some figures about my father's database (on an AMD 3200+ with 1Gb > > mem). > > > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes about > > 3,5 minutes > > - the resulting database is about 100Mb > > - opening that 100Mb database takes roughly 30-40 seconds > > - in people view the clicking on the little triangles is very confusing > > because of the slow feedback (When I showed this to my 84 year old father > > he clicked a few times on a triangle and nothing happened. It took 5 > > seconds before the list unfolded. That is, a surname list with a lot of > > people, which is not unusual if you have a genealogy for your own family. > > Now if you click twice it takes 10 seconds without any visible result.) > > - scrolling in people view using PgUp and PgDn will confuse the display of > > the cursor line (the blue line with the selected person) > > > > The last two items make Gramps simply not-usable for my father. > > > > Would you say that it is the speed (or lack of) that is the primary problem or > the lack of feedback? The lack of feedback I would say. > It may be possible to add some user feedback for operations that are taking a > long time. I am not sure that it will ever be possible to make a database > with 100,000+ records work very quickly in gramps. The View model will always > be slow with that many records. > > Richard > > > |
From: Gerald B. <ger...@gm...> - 2007-01-15 20:16:04
|
The problem with the standard people view and large databases is the tree view. Probably the database could have an additional index or two to take care of the problem. On the other hand, providing an alternative, flat people view might be handy and not just for performance reasons. With a flat view, it would also be possible to active column sorting in the view -- something I know that I would use. On 1/15/07, Richard Taylor <rjt...@th...> wrote: > On Monday 15 January 2007 19:44, Kees Bakker wrote: > > > > Here are some figures about my father's database (on an AMD 3200+ with 1Gb > > mem). > > > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes about > > 3,5 minutes > > - the resulting database is about 100Mb > > - opening that 100Mb database takes roughly 30-40 seconds > > - in people view the clicking on the little triangles is very confusing > > because of the slow feedback (When I showed this to my 84 year old father > > he clicked a few times on a triangle and nothing happened. It took 5 > > seconds before the list unfolded. That is, a surname list with a lot of > > people, which is not unusual if you have a genealogy for your own family. > > Now if you click twice it takes 10 seconds without any visible result.) > > - scrolling in people view using PgUp and PgDn will confuse the display of > > the cursor line (the blue line with the selected person) > > > > The last two items make Gramps simply not-usable for my father. > > > > Would you say that it is the speed (or lack of) that is the primary problem or > the lack of feedback? > > It may be possible to add some user feedback for operations that are taking a > long time. I am not sure that it will ever be possible to make a database > with 100,000+ records work very quickly in gramps. The View model will always > be slow with that many records. > > Richard > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Gramps-users mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-users > |
From: Eero T. <ee...@us...> - 2007-01-15 20:25:35
|
Hi, On Monday 15 January 2007 22:15, Gerald Britton wrote: > The problem with the standard people view and large databases is the > tree view. Probably the database could have an additional index or > two to take care of the problem. On the other hand, providing an > alternative, flat people view might be handy and not just for > performance reasons. > > With a flat view, it would also be possible to active column sorting > in the view -- something I know that I would use. Searching might also be nicer with flat view? - Eero |
From: <ben...@ug...> - 2007-01-15 20:32:29
|
Quoting Gerald Britton <ger...@gm...>: > The problem with the standard people view and large databases is the > tree view. Probably the database could have an additional index or > two to take care of the problem. On the other hand, providing an > alternative, flat people view might be handy and not just for > performance reasons. > > With a flat view, it would also be possible to active column sorting > in the view -- something I know that I would use. In a large database, active column sorting would make gramps slow, as the columns are not indexed, so this is probably not a solution to the problem. Another index or two, when the database is already 100Mb? I am amazed of some of the numbers given here (5Mb Gedcom to 100Mb GRDB...). I suggest we move this discussion to the devel list, and see how best to handle this issues. Like starting with a testcase database of 100.000 people to see/test where the bottlenecks in speed are. Benny ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
From: Richard T. <rjt...@th...> - 2007-01-15 20:26:51
|
Unfortunately it is not that easy. The reason that the treeview was introduced was because it is quicker than the listview. There is an index on the surname field of the person table and it is this index that is used to generate the treeview. Unfortunately because of the design of the gtktreeview it is very difficult to make it work quickly with large databases. I have been working on an alternative implementation that does speed things up but it will still be ponderous with 100,000+ records. Sorting the listview is only credible on very large databases if you have an index on every column that you wish to use as the sort column. Adding indexes significantly slows down database inserts. Experiments that Alex did a while back showed that introducing too many indexes crippled import of GEDCOM databases. Even suspending the indexes and recreating them after the import was problematic. (Alex may remember the details). Filters are another problem area. To be able to filter a table you have to load in every record and compare it to the filter, this can obviously take a very long time if you have 100,000+ records. Regards Richard On Monday 15 January 2007 20:15, Gerald Britton wrote: > The problem with the standard people view and large databases is the > tree view. Probably the database could have an additional index or > two to take care of the problem. On the other hand, providing an > alternative, flat people view might be handy and not just for > performance reasons. > > With a flat view, it would also be possible to active column sorting > in the view -- something I know that I would use. > > On 1/15/07, Richard Taylor <rjt...@th...> wrote: > > On Monday 15 January 2007 19:44, Kees Bakker wrote: > > > Here are some figures about my father's database (on an AMD 3200+ with > > > 1Gb mem). > > > > > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes > > > about 3,5 minutes > > > - the resulting database is about 100Mb > > > - opening that 100Mb database takes roughly 30-40 seconds > > > - in people view the clicking on the little triangles is very > > > confusing because of the slow feedback (When I showed this to my 84 > > > year old father he clicked a few times on a triangle and nothing > > > happened. It took 5 seconds before the list unfolded. That is, a > > > surname list with a lot of people, which is not unusual if you have a > > > genealogy for your own family. Now if you click twice it takes 10 > > > seconds without any visible result.) - scrolling in people view using > > > PgUp and PgDn will confuse the display of the cursor line (the blue > > > line with the selected person) > > > > > > The last two items make Gramps simply not-usable for my father. > > > > Would you say that it is the speed (or lack of) that is the primary > > problem or the lack of feedback? > > > > It may be possible to add some user feedback for operations that are > > taking a long time. I am not sure that it will ever be possible to make a > > database with 100,000+ records work very quickly in gramps. The View > > model will always be slow with that many records. > > > > Richard > > > > ------------------------------------------------------------------------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > your opinions on IT & business topics through brief surveys - and earn > > cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > > Gramps-users mailing list > > Gra...@li... > > https://lists.sourceforge.net/lists/listinfo/gramps-users |
From: Gerald B. <ger...@gm...> - 2007-01-15 20:33:05
|
It might be safe to say that very large implementations need a different approach altogether -- possibly with a separate database engine. Choose the engine you like (mySql, postgre, Oracle, Db2, whatever) create stored procedures optimized for the application along with the indices needed to support them. The client software would then be just concerned with data flow to/from the server and presentation to the user. Unfortunately, I think that this is far beyond the scope of what gramps is trying to accomplish.' I am curious to know if there are other products out there that can comfortably scale to that order of magnitude. On 1/15/07, Richard Taylor <rjt...@th...> wrote: > Unfortunately it is not that easy. The reason that the treeview was introduced > was because it is quicker than the listview. There is an index on the surname > field of the person table and it is this index that is used to generate the > treeview. Unfortunately because of the design of the gtktreeview it is very > difficult to make it work quickly with large databases. > > I have been working on an alternative implementation that does speed things up > but it will still be ponderous with 100,000+ records. > > Sorting the listview is only credible on very large databases if you have an > index on every column that you wish to use as the sort column. Adding indexes > significantly slows down database inserts. Experiments that Alex did a while > back showed that introducing too many indexes crippled import of GEDCOM > databases. Even suspending the indexes and recreating them after the import > was problematic. (Alex may remember the details). > > Filters are another problem area. To be able to filter a table you have to > load in every record and compare it to the filter, this can obviously take a > very long time if you have 100,000+ records. > > Regards > > Richard > > > On Monday 15 January 2007 20:15, Gerald Britton wrote: > > The problem with the standard people view and large databases is the > > tree view. Probably the database could have an additional index or > > two to take care of the problem. On the other hand, providing an > > alternative, flat people view might be handy and not just for > > performance reasons. > > > > With a flat view, it would also be possible to active column sorting > > in the view -- something I know that I would use. > > > > On 1/15/07, Richard Taylor <rjt...@th...> wrote: > > > On Monday 15 January 2007 19:44, Kees Bakker wrote: > > > > Here are some figures about my father's database (on an AMD 3200+ with > > > > 1Gb mem). > > > > > > > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes > > > > about 3,5 minutes > > > > - the resulting database is about 100Mb > > > > - opening that 100Mb database takes roughly 30-40 seconds > > > > - in people view the clicking on the little triangles is very > > > > confusing because of the slow feedback (When I showed this to my 84 > > > > year old father he clicked a few times on a triangle and nothing > > > > happened. It took 5 seconds before the list unfolded. That is, a > > > > surname list with a lot of people, which is not unusual if you have a > > > > genealogy for your own family. Now if you click twice it takes 10 > > > > seconds without any visible result.) - scrolling in people view using > > > > PgUp and PgDn will confuse the display of the cursor line (the blue > > > > line with the selected person) > > > > > > > > The last two items make Gramps simply not-usable for my father. > > > > > > Would you say that it is the speed (or lack of) that is the primary > > > problem or the lack of feedback? > > > > > > It may be possible to add some user feedback for operations that are > > > taking a long time. I am not sure that it will ever be possible to make a > > > database with 100,000+ records work very quickly in gramps. The View > > > model will always be slow with that many records. > > > > > > Richard > > > > > > ------------------------------------------------------------------------- > > > Take Surveys. Earn Cash. Influence the Future of IT > > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > > your opinions on IT & business topics through brief surveys - and earn > > > cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > _______________________________________________ > > > Gramps-users mailing list > > > Gra...@li... > > > https://lists.sourceforge.net/lists/listinfo/gramps-users > |
From: <ben...@ug...> - 2007-01-15 20:59:13
|
Quoting Richard Taylor <rjt...@th...>: > > Filters are another problem area. To be able to filter a table you have to > load in every record and compare it to the filter, this can obviously take a > very long time if you have 100,000+ records. This should not really be the case. If you filter on a .gramps xml text file of 20 Mb with good regex construction, it should not take more than a couple of seconds. I need 3sec to regex a flat text file of 300Mb. It all depends on how you design search and view. Filter now loops over all handles I have the impression, but this can be written for some to loop over indices first, then records. On a database implementation there should be no reason to have a slow application. Let's be honoust, 100.000 records should be peanuts, you never see more than 50 on a single screen. Benny > > Regards > > Richard > > > On Monday 15 January 2007 20:15, Gerald Britton wrote: >> The problem with the standard people view and large databases is the >> tree view. Probably the database could have an additional index or >> two to take care of the problem. On the other hand, providing an >> alternative, flat people view might be handy and not just for >> performance reasons. >> >> With a flat view, it would also be possible to active column sorting >> in the view -- something I know that I would use. >> >> On 1/15/07, Richard Taylor <rjt...@th...> wrote: >> > On Monday 15 January 2007 19:44, Kees Bakker wrote: >> > > Here are some figures about my father's database (on an AMD 3200+ with >> > > 1Gb mem). >> > > >> > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes >> > > about 3,5 minutes >> > > - the resulting database is about 100Mb >> > > - opening that 100Mb database takes roughly 30-40 seconds >> > > - in people view the clicking on the little triangles is very >> > > confusing because of the slow feedback (When I showed this to my 84 >> > > year old father he clicked a few times on a triangle and nothing >> > > happened. It took 5 seconds before the list unfolded. That is, a >> > > surname list with a lot of people, which is not unusual if you have a >> > > genealogy for your own family. Now if you click twice it takes 10 >> > > seconds without any visible result.) - scrolling in people view using >> > > PgUp and PgDn will confuse the display of the cursor line (the blue >> > > line with the selected person) >> > > >> > > The last two items make Gramps simply not-usable for my father. >> > >> > Would you say that it is the speed (or lack of) that is the primary >> > problem or the lack of feedback? >> > >> > It may be possible to add some user feedback for operations that are >> > taking a long time. I am not sure that it will ever be possible to make a >> > database with 100,000+ records work very quickly in gramps. The View >> > model will always be slow with that many records. >> > >> > Richard >> > >> > ------------------------------------------------------------------------- >> > Take Surveys. Earn Cash. Influence the Future of IT >> > Join SourceForge.net's Techsay panel and you'll get the chance to share >> > your opinions on IT & business topics through brief surveys - and earn >> > cash >> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> > _______________________________________________ >> > Gramps-users mailing list >> > Gra...@li... >> > https://lists.sourceforge.net/lists/listinfo/gramps-users > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Gramps-users mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-users > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
From: Richard T. <rjt...@th...> - 2007-01-15 22:31:36
|
On Monday 15 January 2007 20:58, ben...@ug... wrote: > Quoting Richard Taylor <rjt...@th...>: > > Filters are another problem area. To be able to filter a table you have > > to load in every record and compare it to the filter, this can obviously > > take a very long time if you have 100,000+ records. > > This should not really be the case. If you filter on a .gramps xml text > file of > 20 Mb with good regex construction, it should not take more than a couple > of seconds. I need 3sec to regex a flat text file of 300Mb. It all depends > on how you design search and view. Filter now loops over all handles I have > the impression, but this can be written for some to loop over indices > first, then records. There are certainly optimisations to be made but the prime problem is that GtkTree insists on making a call to the underlying datamodel for each entry in the table, as it starts up. I have an implementation of the model that optimises the work done in that call as far as I can see possible (it simply returns the arguments that were passed to it) but it is still 100,000 calls if there are 100,000 records. It is the underlying GtkTree widget that imposes this not gramps. There may be ways around this, but I have not found them yet. > > On a database implementation there should be no reason to have a slow > application. Let's be honoust, 100.000 records should be peanuts, you > never see > more than 50 on a single screen. > Agreed, you can't see them but GtkTree still wants to know about them :-( Regards Richard > Benny > > > Regards > > > > Richard > > > > On Monday 15 January 2007 20:15, Gerald Britton wrote: > >> The problem with the standard people view and large databases is the > >> tree view. Probably the database could have an additional index or > >> two to take care of the problem. On the other hand, providing an > >> alternative, flat people view might be handy and not just for > >> performance reasons. > >> > >> With a flat view, it would also be possible to active column sorting > >> in the view -- something I know that I would use. > >> > >> On 1/15/07, Richard Taylor <rjt...@th...> wrote: > >> > On Monday 15 January 2007 19:44, Kees Bakker wrote: > >> > > Here are some figures about my father's database (on an AMD 3200+ > >> > > with 1Gb mem). > >> > > > >> > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes > >> > > about 3,5 minutes > >> > > - the resulting database is about 100Mb > >> > > - opening that 100Mb database takes roughly 30-40 seconds > >> > > - in people view the clicking on the little triangles is very > >> > > confusing because of the slow feedback (When I showed this to my 84 > >> > > year old father he clicked a few times on a triangle and nothing > >> > > happened. It took 5 seconds before the list unfolded. That is, a > >> > > surname list with a lot of people, which is not unusual if you have > >> > > a genealogy for your own family. Now if you click twice it takes 10 > >> > > seconds without any visible result.) - scrolling in people view > >> > > using PgUp and PgDn will confuse the display of the cursor line (the > >> > > blue line with the selected person) > >> > > > >> > > The last two items make Gramps simply not-usable for my father. > >> > > >> > Would you say that it is the speed (or lack of) that is the primary > >> > problem or the lack of feedback? > >> > > >> > It may be possible to add some user feedback for operations that are > >> > taking a long time. I am not sure that it will ever be possible to > >> > make a database with 100,000+ records work very quickly in gramps. The > >> > View model will always be slow with that many records. > >> > > >> > Richard > >> > > >> > ---------------------------------------------------------------------- > >> >--- Take Surveys. Earn Cash. Influence the Future of IT > >> > Join SourceForge.net's Techsay panel and you'll get the chance to > >> > share your opinions on IT & business topics through brief surveys - > >> > and earn cash > >> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEV > >> >DEV _______________________________________________ > >> > Gramps-users mailing list > >> > Gra...@li... > >> > https://lists.sourceforge.net/lists/listinfo/gramps-users > > > > ------------------------------------------------------------------------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share > > your opinions on IT & business topics through brief surveys - and earn > > cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > > Gramps-users mailing list > > Gra...@li... > > https://lists.sourceforge.net/lists/listinfo/gramps-users > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share > your opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Gramps-users mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-users |
From: <ben...@ug...> - 2007-01-16 08:19:48
|
I suggest to close this thread now. To conclude: 1/Users can expect some radical speedups on the GUI when using GRDB files in the following releases. Some special attention will be given to it. 2/you need a lot of disk space for 100.000 + persons, as we consider speed more important than diskspace, which is cheap nowadays. 3/some operations are inherently slow, like some filters. You should be aware of this. Thanks Gene for bringing this in the attention again due to your question. Benny Quoting Richard Taylor <rjt...@th...>: > On Monday 15 January 2007 20:58, ben...@ug... wrote: >> Quoting Richard Taylor <rjt...@th...>: >> > Filters are another problem area. To be able to filter a table you have >> > to load in every record and compare it to the filter, this can obviously >> > take a very long time if you have 100,000+ records. >> >> This should not really be the case. If you filter on a .gramps xml text >> file of >> 20 Mb with good regex construction, it should not take more than a couple >> of seconds. I need 3sec to regex a flat text file of 300Mb. It all depends >> on how you design search and view. Filter now loops over all handles I have >> the impression, but this can be written for some to loop over indices >> first, then records. > > There are certainly optimisations to be made but the prime problem is that > GtkTree insists on making a call to the underlying datamodel for each entry > in the table, as it starts up. I have an implementation of the model that > optimises the work done in that call as far as I can see possible (it simply > returns the arguments that were passed to it) but it is still 100,000 calls > if there are 100,000 records. It is the underlying GtkTree widget that > imposes this not gramps. > > There may be ways around this, but I have not found them yet. > > >> >> On a database implementation there should be no reason to have a slow >> application. Let's be honoust, 100.000 records should be peanuts, you >> never see >> more than 50 on a single screen. >> > > Agreed, you can't see them but GtkTree still wants to know about them :-( > > Regards > > Richard > > >> Benny >> >> > Regards >> > >> > Richard >> > >> > On Monday 15 January 2007 20:15, Gerald Britton wrote: >> >> The problem with the standard people view and large databases is the >> >> tree view. Probably the database could have an additional index or >> >> two to take care of the problem. On the other hand, providing an >> >> alternative, flat people view might be handy and not just for >> >> performance reasons. >> >> >> >> With a flat view, it would also be possible to active column sorting >> >> in the view -- something I know that I would use. >> >> >> >> On 1/15/07, Richard Taylor <rjt...@th...> wrote: >> >> > On Monday 15 January 2007 19:44, Kees Bakker wrote: >> >> > > Here are some figures about my father's database (on an AMD 3200+ >> >> > > with 1Gb mem). >> >> > > >> >> > > - reading in a 5Mb GEDCOM with 24353 people in 9750 families takes >> >> > > about 3,5 minutes >> >> > > - the resulting database is about 100Mb >> >> > > - opening that 100Mb database takes roughly 30-40 seconds >> >> > > - in people view the clicking on the little triangles is very >> >> > > confusing because of the slow feedback (When I showed this to my 84 >> >> > > year old father he clicked a few times on a triangle and nothing >> >> > > happened. It took 5 seconds before the list unfolded. That is, a >> >> > > surname list with a lot of people, which is not unusual if you have >> >> > > a genealogy for your own family. Now if you click twice it takes 10 >> >> > > seconds without any visible result.) - scrolling in people view >> >> > > using PgUp and PgDn will confuse the display of the cursor line (the >> >> > > blue line with the selected person) >> >> > > >> >> > > The last two items make Gramps simply not-usable for my father. >> >> > >> >> > Would you say that it is the speed (or lack of) that is the primary >> >> > problem or the lack of feedback? >> >> > >> >> > It may be possible to add some user feedback for operations that are >> >> > taking a long time. I am not sure that it will ever be possible to >> >> > make a database with 100,000+ records work very quickly in gramps. The >> >> > View model will always be slow with that many records. >> >> > >> >> > Richard >> >> > >> >> > ---------------------------------------------------------------------- >> >> >--- Take Surveys. Earn Cash. Influence the Future of IT >> >> > Join SourceForge.net's Techsay panel and you'll get the chance to >> >> > share your opinions on IT & business topics through brief surveys - >> >> > and earn cash >> >> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEV >> >> >DEV _______________________________________________ >> >> > Gramps-users mailing list >> >> > Gra...@li... >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-users >> > >> > ------------------------------------------------------------------------- >> > Take Surveys. Earn Cash. Influence the Future of IT >> > Join SourceForge.net's Techsay panel and you'll get the chance to share >> > your opinions on IT & business topics through brief surveys - and earn >> > cash >> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> > _______________________________________________ >> > Gramps-users mailing list >> > Gra...@li... >> > https://lists.sourceforge.net/lists/listinfo/gramps-users >> >> ---------------------------------------------------------------- >> This message was sent using IMP, the Internet Messaging Program. >> >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share >> your opinions on IT & business topics through brief surveys - and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Gramps-users mailing list >> Gra...@li... >> https://lists.sourceforge.net/lists/listinfo/gramps-users > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Gramps-users mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-users > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |