Re: [Postgres-xc-general] tracing cause of datanode crash

[Masataka S. <pg...@gm...>]

Hi,

I think I may be onto something.

When the memory is exhausted, the Linux kernel kills process randomly
by SIGKILL.
The mechanism is called OOM(Out Of Memory) killer.
OOM killer logs its activity to kernel mesage buffer.

Could you check display message (dmesg) and memory resource?

Regards.

On Tue, Nov 12, 2013 at 3:55 AM, Sandeep Gupta <gup...@gm...> wrote:
> Hi Koichi,
>
>    It is a bit of mystery  because it does not happen consistently. Thanks
> for the clarification though. It is indeed helpful.
>
> -Sandeep
>
>
>
>
> On Mon, Nov 11, 2013 at 1:58 AM, 鈴木 幸市 <ko...@in...> wrote:
>>
>> Someone is sending SIGKILL to coordinator/datanode backend processes.
>> Although XC (originally PG) code has a handler for SIGKILL, I didn’t found
>> any code in XC sending SIGKILL to other processes.   I’m afraid there could
>> be another process sending SIGKILL to them.
>>
>> Best Regards;
>> ---
>> Koichi Suzuki
>>
>> 2013/11/09 2:54、Sandeep Gupta <gup...@gm...> のメール：
>>
>> > I am running single instance of postgres-xc with  several datanode
>> > Each data node runs on its own machine.
>> > After instantiating the cluster the database sits idle. I do not perform
>> > any create table or insert operations. However, after some time all the
>> > datanode  automatically shuts down (log messages attached).
>> >
>> > Any pointers as to why this maybe happening would be very useful.
>> >
>> > -Sandeep
>> >
>> > The log files across all datanode contain various error messages. These
>> > include:
>> >
>> >
>> > LOG:  statistics collector process (PID 23073) was terminated by signal
>> > 9: Killed
>> > LOG:  WAL writer process (PID 23071) was terminated by signal 9: Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > HINT:  In a moment you should be able to reconnect to the database and
>> > repeat your command.
>> > LOG:  all server processes terminated; reinitializing
>> >
>> >
>> > LOG:  checkpointer process (PID 21406) was terminated by signal 9:
>> > Killed
>> > LOG:  terminating any other active server processes
>> > WARNING:  terminating connection because of crash of another server
>> > process
>> > DETAIL:  The postmaster has commanded this server process to roll back
>> > the current transaction and exit, because another server process exited
>> > abnormally and possibly corrupted shared memory.
>> > LOG:  statistics collector process (PID 28881) was terminated by signal
>> > 9: Killed
>> > LOG:  autovacuum launcher process (PID 28880) was terminated by signal
>> > 9: Killed
>> > LOG:  terminating any other active server processes
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > November Webinars for C, C++, Fortran Developers
>> > Accelerate application performance with scalable programming models.
>> > Explore
>> > techniques for threading, error checking, porting, and tuning. Get the
>> > most
>> > from the latest Intel processors and coprocessors. See abstracts and
>> > register
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models. Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>