#80 SQLite: SQLSTATE[HY000]: General error: 17 database schema has changed

[Anonymous]

Hi,

I've a very strange bug with SQLite. When I start my crawler in multi-process mode, there is a regular warning: Warning: PDOStatement::execute(): SQLSTATE[HY000]: General error: 17 database schema has changed in /Users/remimichel/PhpstormProjects/crawler_torrent_websites/libs/php-crawl/UrlCache/PHPCrawlerSQLiteURLCache.class.php on line 128

This warning is triggered when I start the crawler from my Mac (Yosemite) and from Ubuntu 14.04 (with PHP 5.3 and 5.5). The strange thing is that it works pretty well when I start it from Ubuntu 12.04 (PHP 5.3.10)

The bad news is that the crawler stops within 1 or 2 minutes (when too many warnings are triggered I guess!! :) )

Do you have any idea? Is it related to my version of PHP?

THanks!

[Anonymous]

Hi!

Waht version of phpcrawl are you using, the current one?

Could you post your setup of the crawler (what multiprocess-mode, number of processes used etc.)?

And does the error occure after every request the crawler does, or just "randomly"?

I remeber i had exaclty the same issue times ago (verion 0.80 or something), but don't know the solution anymore, i'll have to take a look through colsed bugs.

[Anonymous]

Hi

About the version of PHPCrawl, I'm using the last one (0.82). Maybe I should have tried another version!

My configuration is pretty simple:
- $crawler->setUrlCacheType(PHPCrawlerUrlCacheTypes::URLCACHE_SQLITE);
- $crawler->goMultiProcessed(10);
No matter how many processes I setup, I have the problem. I also tried to use URLCACHE_MEMORY for the cache memory with no success!! :(

And the error occures after every request. The most frustrating is that the crawler stops when too many warnings occure.

[Anonymous]

I think I'll have to setup a virtual machine with the exact same settings like your machine (Ubuntu) to reproduce the bug.

So it is a Ubuntu 14.04 standard server-setup (without any non-stanard packages regarding php and sqlite)?

I made some quick tests on different machines avialble here (Ubuntu, Debian, OpenSuse) without any issues.

And i don't know anymore where and why i got this errors too some time ago, didn't find anything in my documentations, somehow i solved it (or was it just a dream ;)?

By the way: phpcrawl ALWAYS uses the slite-urlcache in multiprocess-mode automatically, no matter what cache-type you are setting by setUrlCacheTyp(), doesn't make a difference.

[Anonymous]

Ok, so strange that you did not have the issue again. In fact if I want to avoid the issue, I have to use an Ubuntu 12.04 with standards packages. I do have the issue on Ubuntu 14.04 and on OSX (Yosemite).

Which versions of PHP and SQlite did you try?
Will another version of PhpCrawl be coming out soon?

Thanks so much for your help!

[Anonymous]

Hi again,

I just installed Ubuntu 14 as a virtual machine and tested phpcrawl in multiprocess-mode.
Everything works fine here, i get NO errors.

Here some information about the setup:

OS:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty

PHP:
PHP 5.5.9-1ubuntu4.5 (cli) (built: Oct 29 2014 11:59:10)

SQlite-packages:
libsqlite3-0:amd64 | 3.8.2-1ubuntu2 | amd64
php5-sqlite | 5.5.9+dfsg-1ubuntu4.5 | amd64

I just installed the OS and the two packages "php5" and "php5-sqlite".

As said, it's a VM here, but that shouldn't make any difference i think.

So what can i do?

Another strange thing is that nobody else seems to have this problem, never heared about this since phpcrawl verion 0.8 (or something).

Or is anybody else reading this came acros there errors?

Again, i would like to help you, but right now i don't know how.

[Anonymous]

... and yes, a new verion of phpcrawl will be released probably before christmas (or maybe in the very beginning of the next year).

But it wont change anything regarding your problem i guess (except we'll find a fix together here)

[Uwe Hunfeld]

Just renamed topic.

If anybody else reading this getting the same sqlite-errors, please cry "here".

[Anonymous]

Ok, fine, I thank you very much for your help. At least you tried everything you could to get the bug. I'm a developer and I know how hard it is to reproduce some bugs.

I'll try to reinstall the whole system on another VM. Maybe I'll get lucky this time.

[Anonymous]

I'm not saying that the mystery is solved but I finally succeeded installing my crawler on a new VM (Debian 7 + php54). The crawler is currently working with no bug! ;)

[Anonymous]

I get this error message in my Project since a year of non error use. Is there any known problem with sqlite versions or php version. I use the crawler on a hetzner managed server and it is possible they changed some version. Are there recommende version for sqLite?

[Uwe Hunfeld]

Hi!

Could you please post your version of php and sqlite (and OS) from you hetzner-server? I'm still not able to reproduce this error anywhere.

Maybe we'll gonna figure it out here with some posts from users.

And what version of phpcrawl do you use (before and after the errors occured)?

Thanks!!

[Anonymous]

My php version is 5.4.37 and pdo_sqlite version is 3.7.7.1
I use phpCrawl 0.82
I think they use a modified DEBIAN 3.2.63

[Uwe Hunfeld]

Thanks!

Ok, this is just an an idea:
Would it be possible to get a (strictly limited) access to your hetzner-server (like a user with almost no permissions and just a home-dir for testing)?
Then i could run a few tests with phpcrawl on your server and try to find out what's the problem there and in generel with this error. Maybe we can fix this bug like that.

Otherwise i don't know what to do about it right know, i didn't even see the error live for myself so far.

As i said, just an idea and a question, i understand if you don't want that.

PS: Looks like you are from germany too (Hetzner ;) ), so we could phone if you agree with the idea, just send me an email and i'll send back my phone-number if you want.

Thanks!

[Jan Grubenbecher]

Hallo Uwe,
danke für Deine Mail. Ich konnte den Fehler weiter eingrenzen. Ich denke
nicht, das es am Crawler liegt. Nachdem ich in der URLCache-Klasse das
preparedStatment durch einen direktes exec ersetzt habe ist der Fehler
nicht wieder aufgetreten. Ich weiß das ist keine saubere Lösung, aber
zumindest ein Workaround. Mir scheint, das der laut sqLite-Changelog
vermerkte Bugfix in der Version 3.7.7.1 nicht so ganz korrekt läuft.
Wenn Du weitere Informationen benötigst melde Dich einfach direkt bei mir.

[Jan Grubenbecher]

Sorry, for answering in german. I will post my actual work around tomorrow in english as well.

[Anonymous]

I Have the Same Problem

Using CentOS 6.5
PHP 5.6.5
pdo_sqlite 3.8.7.2

Even the included multiprocessing_example.php gives out this error

[Uwe Hunfeld]

Hi!

Thanks for the report!

Just an question: Any chance to get (striclty restricted) access to that machine for examinig the error and find a solution/workaround for this problem?

[Bento]

vnc access ? sure just tell me when u are online

[Bento]

I am getting the same problem for resumable_example.php

[Uwe Hunfeld]

SSH access possible? I have to be able to upload/modify and run phpcrawl from the commandline on your server to do some tests and hopefully find a fix.

Would it be possible to get limited ssh-access or similar shell?

[Bento]

Will message you the details.

Done, Pls check ur email

[Uwe Hunfeld]

Hi Bento,

got your mail, thanks a lot!
I won't be able to do the tests on your server this weekend, but at the beginning of next week, just you know.

Again, thanks!

[Uwe Hunfeld]

With the help of Bento the problerm could (hopefully) get fixed.

All users having this "SQLite: SQLSTATE[HY000]: General error: 17" error please
try to use the attached "PHPCrawlerSQLiteURLCache.class"-file (and replace the original one in the "libs/UrlCache/"-directory of your phpcrawl package).

And PLEASE let us know here it this fix worked for you!

Thanks!

[Bento]

it works for me

Thanks for the fix