I received the following email from one of my web hosting services today:
Please be advised that your hosting account is in violation of our Internet Service Agreement at due to consistent 73% CPU usage by your website during the last 48 hours. CPU usage must never exceed 33% on shared hosting accounts. We have found the following to be the primary cause of your problem:
Your /genealogy/install.php and other scripts in this directory have been severely overloading and crashing the server. Unfortunately, your website has caused downtime for other clients on your server on multiple occasions, and we must take immediate action to disable your account in order to preserve the quality and integrity of our service. In order to re-activate your account, please confirm that you will correct the above problem within 72 hours from the time of this notice. Thank you for your cooperation.
I just upgraded to 4.2.2 last Friday. My database includes 43k individuals. I've not received any errors. I already had automatic syncing of the GEDCOM disabled, and I reduced my memory limit from 128 to 96 after I got this email.
Unfortunately, this hosting service uses a proprietary backend that doesn't let you see much, and no ssh access.
Any other suggestions?
Thanks,
Matt
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-09
Matt, with 43k individuals you are way too big to be using a shared host, in my view. You will have nothing but problems like this. You need to move.
For the same reason, reducing memory to 96mb won't be great on a GEDCOM that big, although it will still function.
On top of all that, personally I would find having little or no access to the back-end intolerable. I don't need to use it often, but its re-assuring to know I can if I need to.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Matt
Also, they have told you one of the scripts. It would appear, if you did a new installation and import, that your CPU usage would be nearly pegged during the import period, but settle back down under normal use. Frankly, this will occur anytime you load the server with new GEDCOM imports, major structure changes and certain types of searches. I'm with KIWI - get yourself a dedicated server for this large a DB, but your major CPU pressures are mostly likely peaked until you futz with it again.
Stephen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-09
I didn't re-import when I upgraded, so I don't really understand why install.php would be running so much.
Any ideas why this might have happened after the upgrade to 4.2.2
There is backend access, it's just lousy. I manage this site for my Father, and I've been trying to get him to move to bluehost anyway, maybe this will push him a little. He's been with the same people for 10+ years They are so small that he's probably been on the same server the entire time. I've got a PGV installation with bluehost that's almost as big (30k), and always runs like a top.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-09
Interesting, and you've probably already considered what I'm going to say, but why not put his GEDCOM onto your bluehost installation or better still merge the two into one, or are they completely unrelated ? That would seem strange as you are father/son.
It would make your job much easier, only administering one site; and the collaborative benefits are exactly why PGV was created in the first place.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-09
... and I should also add, the reason for so much activity on your site during the upgrade would have been because 4.2.2 will have automatically upgraded and changed some tables in your database, all part of improving the speed and efficiency of PGV. I don't know the technical details, but pretty sure that accessing scripts in install.php (in the background) would have been part of it. So Stephen's comments still apply - these were one-off spikes in your usage.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ummmm, ....
You say that you didn't re-import after the upgrade.
Unfortunately, this is a bad thing. You DO need to re-import, so that the databases will be properly populated, particularly the new columns, as well as columns such as the Soundex, where the algorithm has changed.
I'm with everybody else: move to a more up-to-date hosting service.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-10
We're going to try moving this GEDCOM over to the bluehost.com install. (I'm getting the stats on the existing server, but I'm guessing it's 2 CPUs vs 8 on bluehost.) When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation?
I followed the upgrade guide to the letter, and it said that re-importing was optional when upgrading from the version of PGV I was running. I took that at face value. I'm afraid to do it now, as I would imagine it will max out the CPU.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
<<it said that re-importing was optional when upgrading from the version of PGV I was running>>
You didn't say what version you upgraded from, but if it was 4.2.0 or 4.2.1, then you do not need to reimport.
For the last couple of years, we have adopted the convention that an upgrade that affects only the 3rd digit of a version number does not require a reimport, whereas one that changes the first 2 digits does.
A common cause of poor performance is mismatched collations in your database.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-10
<<When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation? >>
That depends what you are planning to set up on bluehost....
[Whatever you do, take 2 backups of both databases before you start (one to work with, and one put away somewhere safe "just in case")]
1 - If you are creating a completely separate installation (own domain name, files, database etc) then you can import your user-table (and all the others that are not maintained by the GEDCOM file import)
2 - If you are adding the other GEDCOM file as a second GEDCOM on your existing installation, then you can still import the user table, by take care to tell it to append, not replace the data.
3 - if you are merging to a single GEDCOM file, then 2 still applies, but you also need to figure out how to merge the files. Lots of discussion here about that as a starting point for you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
<<<<When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation? >>>>
If the collation settings are the same on both databases/tables/columns, then yes.
If not, you would be better to use the db import/export tool from the patches tracker.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-10
I would rather merge into one GEDCOM, but they are two rather well established sets of researchers that I introduced PGV to, and given the difficulty I've experienced dealing with "their" work, I don't think they would take kindly to me merging the work for their independent families when I am the only common relative. The one I'm having issues with is a pretty dedicated group who really take a lot of ownership in their work, and we've had issues with sharing going back 15 years to our old static website. In addition, I've got way too much media on both sites, and I know that would be a headache.
I will probably just create a new installation so I can maintain URLs, and make it seamless for the users who are almost all very senior citizens. (My Father is pushing 70 and I think he may be one of the junior members.) Will this be significantly more server load than adding the GEDCOM to the existing installation?
The collation is different, so I guess I'll need to use the import/export tool.
I'm going to set this up in a subfolder of an existing domain. When we're sure the server load is OK, and I redirect the DNS to bluehost, can I just edit config.php to change the URL?
If this doesn't work, rather than paying $100US per month for a dedicated server, we've decided to look at getting a business class Internet service to his house, and running our own server for both.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-10
As per usual, you guys are superstars. Thanks for all your help.
Matt
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
"running our own server" is by far the best option. This gives you full control, and lets you give each instance of PhpGedView as much memory as it needs. This is highly recommended.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2009-09-11
We did self-host in the beginning, but that was in the early days of DSL (512k symetric). I was paying $100+ a month and got 10 static IPs with my account so it just made sense.
I just loaded Red Hat on to an old machine. Worked like a charm
Now I've got 30x the bandwidth at less than half the price, but can't get one static IP without going back to $100/mo. :(
Thanks for confirming my suspicions,
Matt
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've done the same with no-ip.org for years on a dozen or more domains, at least until I got a relatively stable IP at home (it hasn't changed in two years).
Running a home server (if allowed by your provider) is pretty easy on any platform and on almost any machine. RAM is always the best investment even with the older processors.
I've used a 8400 PPC mac, a mini-mac and now a MacPro 3.2ghz dual quad (8-core) with 18gb of ram and frankly they all function well (with a significant improvement with the latest).
-Stephen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I received the following email from one of my web hosting services today:
Please be advised that your hosting account is in violation of our Internet Service Agreement at due to consistent 73% CPU usage by your website during the last 48 hours. CPU usage must never exceed 33% on shared hosting accounts. We have found the following to be the primary cause of your problem:
Your /genealogy/install.php and other scripts in this directory have been severely overloading and crashing the server. Unfortunately, your website has caused downtime for other clients on your server on multiple occasions, and we must take immediate action to disable your account in order to preserve the quality and integrity of our service. In order to re-activate your account, please confirm that you will correct the above problem within 72 hours from the time of this notice. Thank you for your cooperation.
I just upgraded to 4.2.2 last Friday. My database includes 43k individuals. I've not received any errors. I already had automatic syncing of the GEDCOM disabled, and I reduced my memory limit from 128 to 96 after I got this email.
Unfortunately, this hosting service uses a proprietary backend that doesn't let you see much, and no ssh access.
Any other suggestions?
Thanks,
Matt
Matt, with 43k individuals you are way too big to be using a shared host, in my view. You will have nothing but problems like this. You need to move.
For the same reason, reducing memory to 96mb won't be great on a GEDCOM that big, although it will still function.
On top of all that, personally I would find having little or no access to the back-end intolerable. I don't need to use it often, but its re-assuring to know I can if I need to.
Matt
Also, they have told you one of the scripts. It would appear, if you did a new installation and import, that your CPU usage would be nearly pegged during the import period, but settle back down under normal use. Frankly, this will occur anytime you load the server with new GEDCOM imports, major structure changes and certain types of searches. I'm with KIWI - get yourself a dedicated server for this large a DB, but your major CPU pressures are mostly likely peaked until you futz with it again.
Stephen
I didn't re-import when I upgraded, so I don't really understand why install.php would be running so much.
Any ideas why this might have happened after the upgrade to 4.2.2
There is backend access, it's just lousy. I manage this site for my Father, and I've been trying to get him to move to bluehost anyway, maybe this will push him a little. He's been with the same people for 10+ years They are so small that he's probably been on the same server the entire time. I've got a PGV installation with bluehost that's almost as big (30k), and always runs like a top.
Interesting, and you've probably already considered what I'm going to say, but why not put his GEDCOM onto your bluehost installation or better still merge the two into one, or are they completely unrelated ? That would seem strange as you are father/son.
It would make your job much easier, only administering one site; and the collaborative benefits are exactly why PGV was created in the first place.
... and I should also add, the reason for so much activity on your site during the upgrade would have been because 4.2.2 will have automatically upgraded and changed some tables in your database, all part of improving the speed and efficiency of PGV. I don't know the technical details, but pretty sure that accessing scripts in install.php (in the background) would have been part of it. So Stephen's comments still apply - these were one-off spikes in your usage.
Ummmm, ....
You say that you didn't re-import after the upgrade.
Unfortunately, this is a bad thing. You DO need to re-import, so that the databases will be properly populated, particularly the new columns, as well as columns such as the Soundex, where the algorithm has changed.
I'm with everybody else: move to a more up-to-date hosting service.
We're going to try moving this GEDCOM over to the bluehost.com install. (I'm getting the stats on the existing server, but I'm guessing it's 2 CPUs vs 8 on bluehost.) When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation?
I followed the upgrade guide to the letter, and it said that re-importing was optional when upgrading from the version of PGV I was running. I took that at face value. I'm afraid to do it now, as I would imagine it will max out the CPU.
<<it said that re-importing was optional when upgrading from the version of PGV I was running>>
You didn't say what version you upgraded from, but if it was 4.2.0 or 4.2.1, then you do not need to reimport.
For the last couple of years, we have adopted the convention that an upgrade that affects only the 3rd digit of a version number does not require a reimport, whereas one that changes the first 2 digits does.
A common cause of poor performance is mismatched collations in your database.
<<When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation? >>
That depends what you are planning to set up on bluehost....
[Whatever you do, take 2 backups of both databases before you start (one to work with, and one put away somewhere safe "just in case")]
1 - If you are creating a completely separate installation (own domain name, files, database etc) then you can import your user-table (and all the others that are not maintained by the GEDCOM file import)
2 - If you are adding the other GEDCOM file as a second GEDCOM on your existing installation, then you can still import the user table, by take care to tell it to append, not replace the data.
3 - if you are merging to a single GEDCOM file, then 2 still applies, but you also need to figure out how to merge the files. Lots of discussion here about that as a starting point for you.
<<<<When we do this can I just export my user table via phpmyadmin and import them into the other installation, or will this overwrite the tables in the existing installation? >>>>
If the collation settings are the same on both databases/tables/columns, then yes.
If not, you would be better to use the db import/export tool from the patches tracker.
I would rather merge into one GEDCOM, but they are two rather well established sets of researchers that I introduced PGV to, and given the difficulty I've experienced dealing with "their" work, I don't think they would take kindly to me merging the work for their independent families when I am the only common relative. The one I'm having issues with is a pretty dedicated group who really take a lot of ownership in their work, and we've had issues with sharing going back 15 years to our old static website. In addition, I've got way too much media on both sites, and I know that would be a headache.
I will probably just create a new installation so I can maintain URLs, and make it seamless for the users who are almost all very senior citizens. (My Father is pushing 70 and I think he may be one of the junior members.) Will this be significantly more server load than adding the GEDCOM to the existing installation?
The collation is different, so I guess I'll need to use the import/export tool.
I'm going to set this up in a subfolder of an existing domain. When we're sure the server load is OK, and I redirect the DNS to bluehost, can I just edit config.php to change the URL?
If this doesn't work, rather than paying $100US per month for a dedicated server, we've decided to look at getting a business class Internet service to his house, and running our own server for both.
As per usual, you guys are superstars. Thanks for all your help.
Matt
"running our own server" is by far the best option. This gives you full control, and lets you give each instance of PhpGedView as much memory as it needs. This is highly recommended.
We did self-host in the beginning, but that was in the early days of DSL (512k symetric). I was paying $100+ a month and got 10 static IPs with my account so it just made sense.
I just loaded Red Hat on to an old machine. Worked like a charm
Now I've got 30x the bandwidth at less than half the price, but can't get one static IP without going back to $100/mo. :(
Thanks for confirming my suspicions,
Matt
<<but can't get one static IP without going back to $100/mo>>
I use dyndns. They will sell you a domain-name, and route the IP to your dynamic IP address at home.
My adsl router (a consumer netgear one) even talks directly to dyndns, so everytime my IP address changes, it updates its DNS servers.
Works a treat.
I like that idea. If you don't mind me asking, What are the specs for the hardware that you use at home?
I've done the same with no-ip.org for years on a dozen or more domains, at least until I got a relatively stable IP at home (it hasn't changed in two years).
Running a home server (if allowed by your provider) is pretty easy on any platform and on almost any machine. RAM is always the best investment even with the older processors.
I've used a 8400 PPC mac, a mini-mac and now a MacPro 3.2ghz dual quad (8-core) with 18gb of ram and frankly they all function well (with a significant improvement with the latest).
-Stephen
<<What are the specs for the hardware that you use at home>>
You don't need much. I've got a 10 year old machine with 512MB of RAM, running the server edition of ubuntu linux - little more than apache and mysql.