Menu
▾
▴
Re: [Assp-test] ASSP fails with no error message when tmpDB folder full
|
From: Colin W. <co...@la...> - 2014-04-19 09:41:09
|
A bit more on this,
Ubuntu 12.04 LTS died with a kernel error during last night's rebuild.
The system logged the following problems with perl 9 times at 07:39, then
waited 2 minutes before logging it a final time before becoming unresponsive
and required a hard reset. All 10 messages are identical bar the first line
where the number beginning with 2 changes " perl D 00002bd3 0
XXXXX 1 0x00000000"
I am going to start testing upgrading 12.04 LTS to 14.04 LTS to get the
newer perl. Hopefully that will help somewhat.
I was thinking about allocating more space to the ramdisk but then
remembered that you recommended to run on an x86 system. An x86 system can't
make use of more than 3.6GB so if I allocate more memory to the ramdisk then
I am severly limiting that which is available to the system.
Are the recommendations still the same?
Sample log of the kernel lockup:
Apr 18 00:09:39 mail2 kernel: [746880.520185] perl D 00002bd3
0 20566 1 0x00000000
Apr 18 00:09:39 mail2 kernel: [746880.520192] e9ed9e20 00200286 0000bb11
00002bd3 00002bd3 c0910fe0 c0a37e00 c0a37e00
Apr 18 00:09:39 mail2 kernel: [746880.520200] a3383ba9 0002a70b ebbe4e00
c175d8d0 d21c32c0 00000000 0000001a e9a100c0
Apr 18 00:09:39 mail2 kernel: [746880.520208] ffffffec 00000000 e9ed9df8
c06aba9d b7585730 00000000 c175d8d0 c16b8000
Apr 18 00:09:39 mail2 kernel: [746880.520225] Call Trace:
Apr 18 00:09:39 mail2 kernel: [746880.520238] [<c06aba9d>] ?
_raw_spin_lock_irqsave+0x2d/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520246] [<c0158e5c>] ?
mm_release+0xdc/0xf0
Apr 18 00:09:39 mail2 kernel: [746880.520250] [<c06a9ea5>]
schedule+0x35/0x50
Apr 18 00:09:39 mail2 kernel: [746880.520254] [<c015eb7d>]
exit_mm+0x6d/0x100
Apr 18 00:09:39 mail2 kernel: [746880.520258] [<c015ed49>]
do_exit+0x139/0x3c0
Apr 18 00:09:39 mail2 kernel: [746880.520263] [<c016bd97>] ?
recalc_sigpending+0x17/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520267] [<c016bf11>] ?
dequeue_signal+0x31/0x190
Apr 18 00:09:39 mail2 kernel: [746880.520271] [<c015f128>]
do_group_exit+0x38/0xa0
Apr 18 00:09:39 mail2 kernel: [746880.520276] [<c016e1d6>]
get_signal_to_deliver+0x1b6/0x3e0
Apr 18 00:09:39 mail2 kernel: [746880.520283] [<c011197f>]
do_signal+0x3f/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520289] [<c0109809>] ?
xen_clocksource_read+0x19/0x20
Apr 18 00:09:39 mail2 kernel: [746880.520293] [<c01848db>] ?
ktime_get_ts+0xeb/0x120
Apr 18 00:09:39 mail2 kernel: [746880.520300] [<c0257514>] ?
poll_select_set_timeout+0x64/0x80
Apr 18 00:09:39 mail2 kernel: [746880.520304] [<c025829a>] ?
sys_poll+0x5a/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520308] [<c0111c25>]
do_notify_resume+0x75/0x90
Apr 18 00:09:39 mail2 kernel: [746880.520313] [<c06abd10>]
work_notifysig+0x13/0x1b
-----Original Message-----
From: Colin Waring [mailto:co...@la...]
Sent: 16 April 2014 09:21
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full
Hi Thomas,
ASSP died again overnight due to tmpDB being full. It looks like
BDBMaxCacheSize doesn't prevent ASSP from dying but allows it to clear the
folder and start up again once it does.
All the best,
Colin Waring.
-----Original Message-----
From: Thomas Eckardt [mailto:Tho...@th...]
Sent: 11 April 2014 08:12
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full
Has anything changed recently that would increase the tmpDB requirements?
Thins could happen - it depends on the config and the count of files and
words in the corpus.
How ever 1GB for tmpDB is also too less for my system.
There are some improvements for BDB cache calculation in the latest
versions. This cache settings are useless for systems that uses a RAM-drive
for tmpDB - I'll cange this.
Thomas
Von: "Colin Waring" <co...@la...>
An: "'ASSP development mailing list'"
<ass...@li...>,
Datum: 11.04.2014 08:51
Betreff: Re: [Assp-test] ASSP fails with no error message when
tmpDB folder full
Hi Thomas,
I didn't have to wait long, it turns out two runs of rebuildspamdb are
enough to fill a 1GB tempdb folder now.
I have added the entry and ASSP starts back up without coredumping. tmpDB
does remain 100% full though
Has anything changed recently that would increase the tmpDB requirements?
All the best,
Colin Waring.
-----Original Message-----
From: Thomas Eckardt [mailto:Tho...@th...]
Sent: 09 April 2014 09:23
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full
add the following line to 'lib/CorrectASSPcfg.pm'
$main::BDBMaxCacheSize = 0;
and restart assp. Tell me if it works or not.
Thomas
Von: "Colin Waring" <co...@la...>
An: "'ASSP development mailing list'"
<ass...@li...>,
Datum: 09.04.2014 09:56
Betreff: [Assp-test] ASSP fails with no error message when tmpDB
folder full
Hi Folks,
At the weekend one of my mailservers died overnight. I had a quick check
over and saw that it was coredumping without any errors. I decided to
leave
it till the morning as the other servers could handle things. By morning
my
monitoring scripts had restarted it.
This morning I got up to the same issue, except the problem didn't go away
itself.
ASSP would core dump during startup without outputting any messages. If I
enabled debugging, not debug file was created.
I had to resort to strace to see that it was getting an error with space
on
the tmpDB folder which was indeed completely full. There was over a
gigabyte
of data contained in there, all relating to rebuildspamdb. Some was from
last night's run but some was from two nights prios.
I'm presuming that there is some code in the startup that clears up
leftover
rebuildspamdb data as I emptied the folder but did not remove it. After
starting ASSP the rebuild folder disappeared.
I suspect this code needs to be called much earlier in the startup
process.
I'll include the strace failure incase it gives an idea of where in the
process it needs to be moved to.
All the best,
Colin Waring.
stat64("/usr/local/assp/tmpDB", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80,
...}) = 0
stat64("/usr/local/assp/tmpDB/_cachecheck", {st_mode=S_IFDIR|0755,
st_size=80, ...}) = 0
lstat64("/usr/local/assp/tmpDB/_cachecheck/__db.001",
{st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
unlink("/usr/local/assp/tmpDB/_cachecheck/__db.001") = 0
lstat64("/usr/local/assp/tmpDB/_cachecheck/__db.002", 0x8c07064) = -1
ENOENT
(No such file or directory)
lstat64("/usr/local/assp/tmpDB/_cachecheck/__db.003", 0x8c07064) = -1
ENOENT
(No such file or directory)
lstat64("/usr/local/assp/tmpDB/_cachecheck/__db.004", 0x8c07064) = -1
ENOENT
(No such file or directory)
lstat64("/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt",
{st_mode=S_IFREG|0644, st_size=98, ...}) = 0
unlink("/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt") =
0
open("/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt",
O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4
_llseek(4, 0, [0], SEEK_END) = 0
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY
(Inappropriate ioctl for device)
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
time(NULL) = 1397029108
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
time(NULL) = 1397029108
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 5
read(5, "0\n", 8192) = 2
close(5) = 0
write(4, "2014-04-09 08:38:28\nBDB cachesiz"..., 50) = 50
fcntl64(4, F_GETFL) = 0x8401 (flags
O_WRONLY|O_APPEND|O_LARGEFILE)
fstat64(4, {st_mode=S_IFREG|0644, st_size=50, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
=
0xb6ee0000
_llseek(4, 0, [50], SEEK_CUR) = 0
open("/usr/local/assp/tmpDB/_cachecheck/DB_CONFIG", O_RDONLY|O_LARGEFILE)
=
-1 ENOENT (No such file or directory)
stat64("/var/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
open("/usr/local/assp/tmpDB/_cachecheck/__db.001",
O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 5
fcntl64(5, F_GETFD) = 0
fcntl64(5, F_SETFD, FD_CLOEXEC) = 0
open("/usr/local/assp/tmpDB/_cachecheck/__db.001",
O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8
fcntl64(8, F_GETFD) = 0
fcntl64(8, F_SETFD, FD_CLOEXEC) = 0
_llseek(8, 16384, [16384], SEEK_SET) = 0
write(8,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
8192)
= -1 ENOSPC (No space left on device)
write(4, "write: 0xbc3caf0, 8192: No space"..., 48) = 48
mmap2(NULL, 24576, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0) = 0xb66bd000
close(8) = 0
--- SIGBUS (Bus error) @ 0 (0) ---
+++ killed by SIGBUS (core dumped) +++
--------------------------------------------------------------------------
--
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Assp-test mailing list
Ass...@li...
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
--------------------------------------------------------------------------
--
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
_______________________________________________
Assp-test mailing list
Ass...@li...
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
--------------------------------------------------------------------------
----
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Assp-test mailing list
Ass...@li...
https://lists.sourceforge.net/lists/listinfo/assp-test
|
