Re: [Postgres-xc-developers] SEMOP PANIC in datanode causes Postgres XC to freeze

[Koichi S. <koi...@gm...>]

It is system level error which may not happen so often in production.   The
cause is failure in semaphore operation through Linux system call called
semop().   The failure may be called if you leave too many semaphore
unreleased in your system.

If you kill too many postmaster process of coordinator/datanode, they may
lose a chance to release these kernel resourcdes.

Please try ipcs -s (shows semaphore) and ipcs -m (shared memory) for all
the users in your system.    It may help to see what's going on.


----------
Koichi Suzuki


2013/4/24 Venky Kandaswamy <ve...@ad...>

>  Hi All,
>
>     One of the datanodes (but not always the same node) gets a PANIC from
> the kernel and aborts. This causes the entire system to freeze. Cannot get
> any new connections and some transactions go into waiting. The message we
> are seeing in the logs are cryptic:
>
> {time user process message}
>
> 2013-04-23 11:42:49 PDT adchemy 17573 PANIC:  semop(id=189366400) failed:
> Invalid argument
> 2013-04-23 11:42:49 PDT adchemy 17561 PANIC:  semop(id=189366400) failed:
> Invalid argument
> 2013-04-23 11:42:50 PDT adchemy 17563 PANIC:  semop(id=189366400) failed:
> Invalid argument
> 2013-04-23 11:42:50 PDT adchemy 17576 PANIC:  semop(id=189366400) failed:
> Invalid argument
> 2013-04-23 12:08:31 PDT adchemy 17801 PANIC:  semop(id=203915392) failed:
> Invalid argument
> 2013-04-23 12:08:31 PDT adchemy 17789 PANIC:  semop(id=203915392) failed:
> Invalid argument
> 2013-04-23 12:08:31 PDT adchemy 17787 PANIC:  semop(id=203915392) failed:
> Invalid argument
> 2013-04-23 12:08:33 PDT adchemy 17803 PANIC:  semop(id=203915392) failed:
> Invalid argument
> 2013-04-23 12:08:33 PDT adchemy 17791 PANIC:  semop(id=203915392) failed:
> Invalid argument
> 2013-04-23 12:09:31 PDT analytics 17812 PANIC:  semop(id=203882623)
> failed: Invalid argument
> 2013-04-23 12:09:31 PDT adchemy 17797 PANIC:  semop(id=203882623) failed:
> Invalid argument
> 2013-04-23 12:14:11 PDT analytics 17785 PANIC:  semop(id=203882623)
> failed: Invalid argument
> 2013-04-23 12:14:11 PDT adchemy 17805 PANIC:  semop(id=203882623) failed:
> Invalid argument
> 2013-04-23 12:16:11 PDT analytics 17783 PANIC:  semop(id=203915392)
> failed: Invalid argument
> 2013-04-23 12:16:11 PDT analytics 17784 PANIC:  semop(id=203915392)
> failed: Invalid argument
> 2013-04-23 12:16:11 PDT analytics 17786 PANIC:  semop(id=203915392)
> failed: Invalid argument
> 2013-04-23 12:23:32 PDT analytics 17938 PANIC:  semop(id=210141312)
> failed: Invalid argument
> 2013-04-23 12:23:32 PDT analytics 18544 PANIC:  semop(id=210141312)
> failed: Invalid argument
> 2013-04-23 12:23:32 PDT analytics 17937 PANIC:  semop(id=210141312)
> failed: Invalid argument
> 2013-04-23 12:23:32 PDT analytics 17951 PANIC:  semop(id=210141312)
> failed: Invalid argument
>
>     Any thoughts on why this might happen?
>
>  ________________________________________
>
> Venky Kandaswamy
>
> Principal Engineer, Adchemy Inc.
>
> 925-200-7124
>
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>