[Postgres-xc-developers] SEMOP PANIC in datanode causes Postgres XC to freeze

[Venky K. <ve...@ad...>]

Hi All,

    One of the datanodes (but not always the same node) gets a PANIC from the kernel and aborts. This causes the entire system to freeze. Cannot get any new connections and some transactions go into waiting. The message we are seeing in the logs are cryptic:

{time user process message}

2013-04-23 11:42:49 PDT adchemy 17573 PANIC:  semop(id=189366400) failed: Invalid argument
2013-04-23 11:42:49 PDT adchemy 17561 PANIC:  semop(id=189366400) failed: Invalid argument
2013-04-23 11:42:50 PDT adchemy 17563 PANIC:  semop(id=189366400) failed: Invalid argument
2013-04-23 11:42:50 PDT adchemy 17576 PANIC:  semop(id=189366400) failed: Invalid argument
2013-04-23 12:08:31 PDT adchemy 17801 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:08:31 PDT adchemy 17789 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:08:31 PDT adchemy 17787 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:08:33 PDT adchemy 17803 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:08:33 PDT adchemy 17791 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:09:31 PDT analytics 17812 PANIC:  semop(id=203882623) failed: Invalid argument
2013-04-23 12:09:31 PDT adchemy 17797 PANIC:  semop(id=203882623) failed: Invalid argument
2013-04-23 12:14:11 PDT analytics 17785 PANIC:  semop(id=203882623) failed: Invalid argument
2013-04-23 12:14:11 PDT adchemy 17805 PANIC:  semop(id=203882623) failed: Invalid argument
2013-04-23 12:16:11 PDT analytics 17783 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:16:11 PDT analytics 17784 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:16:11 PDT analytics 17786 PANIC:  semop(id=203915392) failed: Invalid argument
2013-04-23 12:23:32 PDT analytics 17938 PANIC:  semop(id=210141312) failed: Invalid argument
2013-04-23 12:23:32 PDT analytics 18544 PANIC:  semop(id=210141312) failed: Invalid argument
2013-04-23 12:23:32 PDT analytics 17937 PANIC:  semop(id=210141312) failed: Invalid argument
2013-04-23 12:23:32 PDT analytics 17951 PANIC:  semop(id=210141312) failed: Invalid argument

    Any thoughts on why this might happen?


________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124