4.1 ... large sortable tables fail to render & sort
Brought to you by:
canajun2eh,
yalnifj
Menu
▾
▴
#1455 4.1 ... large sortable tables fail to render & sort
Using anything in the 4.1 branch which features the sortable tables (at the time of writing SVN 1256), any pages that try and deliver large sortable tables either fail to render, or if they do render, they fail to sort.
This is especially apparent with IE6 and IE7. Using FireFox or Opera some of my pages take around 200 seconds to show, which is far far too long.
This has been tried on various servers and it's not a server issue, it appears to be the browser choking on the data that's it's asked for and it can't cope with the result.
For instance, try drilling down the placelist on my test site
http://dxradio.demon.co.uk/pgvsvn/phpgedview/placelist.php choosing such links as "England", then "Cornwall" then "St Just in Penwith" and then choosing the "View all records found in this place" link.
Even if a user happens to be using a browser that "works", a wait of over three minutes for the page to render is unacceptable, and what's the point in a sortable table if the browser is unable to sort it?
Mark
Logged In: YES
user_id=1198414
Originator: NO
Mark supplied me with a copy of his GEDCOM, and we tried this on my server. We observed the same result -- EXTREMELY slow page rendering.
Opera tells us that the "all records for St. Just in Penwith" page is over 12 Mb. This is far too large. It takes a LONG time to send 12 Mb over the Internet.
We need to find a method similar to the way large Indi and Family lists are handled -- The "all records" list is really an Indi list followed by a Family list.
Logged In: YES
user_id=1466942
Originator: NO
This list has ~5000 INDI records and ~2000 fam records. The resultant file is ~12MB in size and downloads at a pretty constant 40kb/sec, taking ~300 seconds.
The DNS for dxradio.demon.co.uk gives an address in the 80.176.144.0 - 80.176.159.255 range, which appears to belong to demon's ADSL residential customers. From this I guess you are running a server at home using your broadband line, rather than a hosted server at an ISP.
In my experience, 40kb/sec upload speed for broadband is pretty much par for the course. Demon claim "up to 448kb/sec upload", but this is dependent on both line condition and contention with other users. I live a few hundred metres from my local exchange, and I rarely get above 30kb/sec.
From this it looks like the time to download is limited by bandwidth, not server IO/CPU/RAM.
Presumably you can access PGV on your server using 127.0.0.1. What is performance like then?
There are some server setting which compress data when sending. (They are in the docs somewhere - I haven't looked at them for a while). This may help.
Logged In: YES
user_id=623181
Originator: YES
Yes, this machine is my "home" PC and is clearly subject to the uplink ability of ADSL fro external users. However, exactly the same end result occurs when I run the browser on either the same machine or on another machine on the home intranet (1 Gbit full duplex)
Specs of machine ... 2.4 GHz Core2Duo, 2 GB RAM, Apache 2.2.4, MySQL 5.0.41 php 5.2.3
Even on the same machine, IE7 completely fails to render the large pages.
I don't know what spec machine Gerry tried this on, but his results were the same.
Mark
Logged In: YES
user_id=1466942
Originator: NO
<<the same end result occurs when I run the browser on either the same machine or on another machine on the home intranet>>
Sorry to labour the point, but when you access PGV locally, what URL do you use?
If you use "dxradio.demon.net/phpgedview", your connection will *still* route out to demon and back again via your slow uplink connection, even though you are connecting to a local machine.
If you are on another machine on your local network, you will need to use a local IP address, probably something like "http://192.168.?.?/phpgedview".
If you are on the server itself, you can use "http://127.0.0.1/phpgedview".
Either way, make sure you don't have $SERVER_URL set in your config.
Logged In: YES
user_id=623181
Originator: YES
Accessing the page *locally* I'm using http://127.0.0.1/pgvsvn/phpgedview ... or from another machine on the home intranet I'm using http://192.168.x.y/ [blah blah].
At the end of the 11.8 MB placelist.php file that results if all I do is right click and "Save Target as", I see the stats shown as
Execution time: 72.683 sec. Total Database Queries: 8065. Total privacy checks: 8053. Total Memory Usage: 52814.95 KB.
On the same machine in IE7, downloading the file to the Desktop rather than into the Browser window, it downloads at about 176 kb/sec.
And this resultant file loads into the browser fine ... *IF* ... it's loaded from somewhere else on the machine, eg when I've saved it to the Desktop. This sort of proves the browser can handle 12 MB of html. However, when loaded into the browser from the Desktop, it doesn't run any JavaScript to produce the sortable columns (and also doesn't load images etc etc)
Mark
Logged In: YES
user_id=623181
Originator: YES
Just for experiment, I've put the "saved" placelist file in the webdirectory ... once as saved, once with the sortable table .js's removed (or rather changed to .jss so they don't load)
http://dxradio.demon.co.uk/pgvsvn/phpgedview/placelist_no_sort.htm
http://dxradio.demon.co.uk/pgvsvn/phpgedview/placelist_sort.htm
on the local machine using http://127......
With Safari, the _no_sort.htm loads pretty quickly and gives me a page that I can scroll up and down. But the _sort.htm version doesn't finish rendering.
With IE7, the _no_sort.htm says "done" eventually, but doesn't scroll ... just eggtimer. The _sort.htm doesn't finish rendering.
I don't believe we're talking about how well the server is serving up pages, we're talking about how well or otherwise browsers use the pages they're given. And the problem is that IE6 and IE7 especially are having big problems, and even more compliant browsers take far too long.
If you want to compare how PGV 3.3.9 works try
http://dxradio.demon.co.uk/familytree/
Mark
Logged In: YES
user_id=1466942
Originator: NO
<<I'm using http://127.0.0.1/>>
Sorry for doubting you, but you'll understand why I wanted to confirm this.
I've been doing the same sort of testing as you (save to local file in/out of PGV tree), and have come to similar conclusions. The rendering speed is only slightly faster than the network speed, hence we've identified one bottleneck, only to find another.
I'll take another look at this later, but one suggestion to speed it up is to replace the external male/female .GIF files with the unicode symbols for male/female. (see http://en.wikipedia.org/wiki/Gender_symbol\)
Logged In: YES
user_id=623181
Originator: YES
In fact I've already added width="14" height="14" to all the /images/small/male.gif and /images/small/female.gif occurences ... but that didn't make any perceptible difference in speed.
The three .js's I've disabled in the _no_sort.htm version (by renaming thus)
<script type="text/javascript" src="strings.jss"></script>
<script type="text/javascript" src="js/kryogenix/sorttable.jss"></script>
<script type="text/javascript" src="js/kryogenix/sorttable_filter.jss"></script>
make a lot more difference in allowing the page to render at all.
However, I'm yet to end up with a "working" sortable version of the page ... even if it finishes rendering the initial view (which apparently is reverse name order !?!), I can't successfully sort it.
With the saved .htm version of the page, I'm obviously bypassing any MySQL or php involvement. Which leaves Javascript processing and html rendering by the browser(s) as the most likely culprits.
Mark
Logged In: YES
user_id=623181
Originator: YES
Also, my main Internet site is at http://www.hattam.co.uk/ ... I'm only using the local machine here as a test bed. But if it doesn't work well "locally", then I can't expect my main site to work well either.
Mark
Logged In: YES
user_id=1006499
Originator: NO
Mark,
you are right : placelist may return (too) big pages. Loading more than 500 lines is too long.
Greg has recently worked on speeding up table rendering (better date calculation, etc...).
Some points I'm working on for 4.2:
- filtering by BIRT/MARR/DEAT to reduce table size [1649660] (done)
- upgrading to much faster kryogenix sorttable.js V2 (in progress)
We may also use paging to serve data by 20 or 50 lines at a time.
A few months ago I've been looking to some frameworks to do this
such as turbogrid
http://www.turboajax.com/products/turbogrid/demo/
About initial view : no sort is done, data order comes from db request.
Christophe
Logged In: YES
user_id=1466942
Originator: NO
FYI, a quick test shows that replacing the male.gif/female.gif images with their unicode characters does save a bit of time. On my test system, the total time to list ~5000 individuals reduces from 120 seconds to 110 seconds. Not major, but every 10% helps :-)
Logged In: YES
user_id=1198414
Originator: NO
Christophe:
Since the Indi and the Family tables are in server memory before they are sent to the Browser, perhaps we should sort them into Name sequence on the server first.
I think Name sequence would be the most commonly desired sequence, with Date being the other.
Logged In: YES
user_id=1006499
Originator: NO
Greg : using unicode chars is a great idea
OK on my XP system using IE of FF
but dont work with Opera...
And what about *nix or Mac OS ?
http://www.alanwood.net/unicode/miscellaneous_symbols.html
Logged In: YES
user_id=1006499
Originator: NO
Other things we can do :
- make some cols optional : anniversary, children count...
- or do not show them when table size > 400 rows
Logged In: YES
user_id=623181
Originator: YES
How would you know in advance when there will be more than 400 rows?
Do Javascript sortable tables have that much of a benefit over running MySQL queries? I know the theory is that you only send the data once from the server to the client, but now we end up having to trust the browser to get it right.
Even when IE6 or IE7 does eventually load a large table, it's still unable to sort it using Javascript.
Also why do we think it necessary to have a number of children column at all in a placelist?
Lastly (for now), the anniversary column for births when presented with a birth year of say 1900 gives a 106-107 entry ... it can only be 107 ...
Mark
Logged In: YES
user_id=1006499
Originator: NO
<<How would you know in advance when there will be more than 400 rows?>>
db query sends array to php function print_indi_table to generate html table
Here we know size of array.
<<Also why do we think it necessary to have a number of children column at
all in a placelist?>>
indilist/famlist/placelist all use same php function to print indi table.
It can be useful to sort any indilist by children count.
I found some errors in my tree using this feature (children linked to a wrong family...)
<<Lastly (for now), the anniversary column for births when presented with a
birth year of say 1900 gives a 106-107 entry ... it can only be 107 ...>>
When only birthyear is known, age is a range.
For someone born 24 dec 1900, age it today 106 not yet 107 ;-)
Logged In: YES
user_id=623181
Originator: YES
ah ... it's an "age (now)" not an "anniversary (this year)" column ... sorry.
At what stage then would columns become optional, or sortable-optional?
Mark
Logged In: YES
user_id=1198414
Originator: NO
Mark:
The print_indi_table() and print_fam_table() functions could be modified to accept optional input parameters that determine which columns are printed (or not), and which columns are sortable (or not).
This shouldn't be hard to implement, but how do we determine which columns are of interest? As Christophe has pointed out, the "Number of Children" column was, on at least one occasion, of interest to him.
I don't think we can pre-determine which columns should be dropped or changed to non-sortable. Perhaps we can let the user determine this, by providing an Option page before the actual Placelist is presented. That way, the actual presentation of the Placelist table can vary case-by-case.
Logged In: YES
user_id=1466942
Originator: NO
The call to the js function "make tables sortable" takes 3-4 seconds on my machine. Probably because it has to scan all 12MB to find the tables. Since all it does is add some code to the table headers, we could do it in PHP more quickly/easily.
Similarly, we could identify the BCOL and DCOL columns manually, so the alive-in-year filter wouldn't need to search for tables/columns. This would probably save another 3-4 seconds.
The "compare" helper function ts_sort_date() looks horribly inefficient. Changes here would have a significant impact on sorting performance.
Logged In: YES
user_id=623181
Originator: YES
again that presupposes users know that the consequence of a large table is ... a search bot won't be able to make choices.
at the moment a non-rendering webpage forces the user to end the application process, which is less than elegant. MacOSX and XPPro do seem to perform this OK, but again it's a burden on the user who we assume knows what to do.
How does the "number of children" lead you to know which families are mis-entered? OK a value of "30" might be an indication, but all you'd need for that is a select max(total_children_in_a_family) sql type query, not to rely on a placelist sorting.
Mark
Logged In: YES
user_id=1006499
Originator: NO
SVN 1260 :
changes to functions_print_list :
- anniversary/children/lastchange cols and indi sex img not printed when table > 300 rows
This should help the "page rendering" step.
About sorting : as js wont be able to sort 2000 rows table in a "short" time,
I suggest to disallow sorting in this case.
Logged In: YES
user_id=623181
Originator: YES
SVN seems unavailable this evening ...
C:\svn co https://svn.sourceforge.net/svnroot/phpgedview/trunk c:\pgvsvn
svn: PROPFIND request failed on '/svnroot/phpgedview/trunk'
svn: PROPFIND of '/svnroot/phpgedview/trunk': Could not resolve hostname `svn.sourceforge.net': The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for. (https://svn.sourceforge.net)
Mark
Logged In: YES
user_id=634811
Originator: NO
The URL changed (search the forums). It is now:
https://phpgedview.svn.sourceforge.net/svnroot/phpgedview/trunk (note the added phpgedview.)
Logged In: YES
user_id=634811
Originator: NO
Christophe,
You mentioned that some performance enhancements were done on the 4.2 branch. These are:
- filtering by BIRT/MARR/DEAT to reduce table size [1649660] (done)
- upgrading to much faster kryogenix sorttable.js V2 (in progress)
Wouldn't it make sens to copy them over to 4.1 since it is a real impact to some users? How much of a performance enhancement do these provide?
Logged In: YES
user_id=623181
Originator: YES
Thanks ... made that change ... but now it whinges that the URL has changed for the local repository and still wont update. I presume it's got the URL written somewhere in the .svn directory.
So I guess I'll just blow away the old local directory, and let it re-sync as if new.
Mark