I have a clean installation of GT.M (GT.M V5.3-000 Linux x86) with freshly created database files.
When accessing the database I randomly get errors like the following:
150372970,indexTags+12^node,%GTM-E-GVPUTFAIL, Global variable put failed. Failure code: CCCC.,%GTM-I-GVIS, Global variable: ^nodex("*","2008-04-15T21:54:17Z","baccdaccabacbda",26044556)
This typically happens after one or two hours of heavy database inserts. It's not inherent database corruption as I've tried recreating the database so that I know it is clean.
I've had GT.M running the same application on this machine previously and not experienced any problems.
I'm beginning to suspect a hardware fault, but just wondered if anyone recognizes this error as a symptom of something else?
George
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I dont remember seeing this before. Assuming the hardware is fine, there are two possibilities.
1) The GT.M process gives an error even though the database is alright. Probably because something in its private memory is incorrect.
2) The GT.M process gives an error because the database has integrity errors.
First one needs to find out which of the above two is happening. Once you get this error, try running a MUPIP INTEG on this database file. If it is clean, then it is (1). If not it is (2). Knowing this will help in further analysis.
The next step is to start a fresh run with a clean database with VIEW "GDSCERT":1 enabled in all GT.M processes accessing the database. This should be done at GT.M process startup. This will cause every update by GT.M to validate the contents of the block. This way the moment the integrity error occurs one will get a core file to analyse.
Note that the VIEW "GDSCERT" scheme involves checking the integrity of every block that gets updated so it might slow things down a little.
Running with a debug build of GT.M (-g option in C compiler at compile and link time enabled) will really help with troubleshooting this.
Also, you can look at syslog to see if there are any messages either from GT.M or from the hardware pointing to any issues. All GT.M messages will contain "GTM-" in them.
Hope this helps. Do let us know your findings.
Narayanan.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
%GTM-F-GTMASSERT, GT.M V5.3-000 Linux x86 - Assert failed /usr/library/V53000/src/cert_blk.c line 342
%GTM-F-GTMASSERT, GT.M V5.3-000 Linux x86 - Assert failed /usr/library/V53000/src/cert_blk.c line 375
%GTM-E-DBRNDWN, Error during global database rundown for region NODEX.
Notify those responsible for proper database operation.
No sign of a core dump. Where should I look to find that?
George
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Core files, if at all they got created, would be in the same directory where the GT.M process started. If not, dumping of core files need to be enabled first.
Depending on your shell, try the "limit" or "ulimit -a" command. You should see "coredumpsize" as unlimited. Someone more familiar with Linux sysadmin will have more ideas on enabling this.
Assuming core dumps are first enabled, would it be possible for you to try the same experiment again with a debug build of GT.M V5.3-000 (I dont know if you are already running one or not)? If you can build it yourself, that would be great. If not, we can provide you with a debug binary distribution. Debugging the core file is a lot easier with debug builds.
Narayanan.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have a clean installation of GT.M (GT.M V5.3-000 Linux x86) with freshly created database files.
When accessing the database I randomly get errors like the following:
150372970,indexTags+12^node,%GTM-E-GVPUTFAIL, Global variable put failed. Failure code: CCCC.,%GTM-I-GVIS, Global variable: ^nodex("*","2008-04-15T21:54:17Z","baccdaccabacbda",26044556)
This typically happens after one or two hours of heavy database inserts. It's not inherent database corruption as I've tried recreating the database so that I know it is clean.
I've had GT.M running the same application on this machine previously and not experienced any problems.
I'm beginning to suspect a hardware fault, but just wondered if anyone recognizes this error as a symptom of something else?
George
I dont remember seeing this before. Assuming the hardware is fine, there are two possibilities.
1) The GT.M process gives an error even though the database is alright. Probably because something in its private memory is incorrect.
2) The GT.M process gives an error because the database has integrity errors.
First one needs to find out which of the above two is happening. Once you get this error, try running a MUPIP INTEG on this database file. If it is clean, then it is (1). If not it is (2). Knowing this will help in further analysis.
The next step is to start a fresh run with a clean database with VIEW "GDSCERT":1 enabled in all GT.M processes accessing the database. This should be done at GT.M process startup. This will cause every update by GT.M to validate the contents of the block. This way the moment the integrity error occurs one will get a core file to analyse.
Note that the VIEW "GDSCERT" scheme involves checking the integrity of every block that gets updated so it might slow things down a little.
Running with a debug build of GT.M (-g option in C compiler at compile and link time enabled) will really help with troubleshooting this.
Also, you can look at syslog to see if there are any messages either from GT.M or from the hardware pointing to any issues. All GT.M messages will contain "GTM-" in them.
Hope this helps. Do let us know your findings.
Narayanan.
Narayanan
mupip integ reports lots of errors so it's not a memory problem with the process.
I've restarted with a clean empty database and enabled GDSCERT. I'll let you know what I find.
George
Narayanan
After a while I got this:
%GTM-F-GTMASSERT, GT.M V5.3-000 Linux x86 - Assert failed /usr/library/V53000/src/cert_blk.c line 342
%GTM-F-GTMASSERT, GT.M V5.3-000 Linux x86 - Assert failed /usr/library/V53000/src/cert_blk.c line 375
%GTM-E-DBRNDWN, Error during global database rundown for region NODEX.
Notify those responsible for proper database operation.
No sign of a core dump. Where should I look to find that?
George
George,
Core files, if at all they got created, would be in the same directory where the GT.M process started. If not, dumping of core files need to be enabled first.
Depending on your shell, try the "limit" or "ulimit -a" command. You should see "coredumpsize" as unlimited. Someone more familiar with Linux sysadmin will have more ideas on enabling this.
Assuming core dumps are first enabled, would it be possible for you to try the same experiment again with a debug build of GT.M V5.3-000 (I dont know if you are already running one or not)? If you can build it yourself, that would be great. If not, we can provide you with a debug binary distribution. Debugging the core file is a lot easier with debug builds.
Narayanan.
For the record, this was a hardware problem. Some memory had become unseated and was behaving unreliably.
Not a GT.M problem at all.