Re: [Sedna-discussion] Sedna 3.5.615 repeated hanging

[Ivan L. <la...@gm...>]

Hi Ilya,

The patch has been applied to the night build 3.6.519, the data has been
migrated with se_exp using complete export-import. The issue has not appear
again for a week already. I'll keep monitoring Sedna.

However, it seems there is another issue with that night build:

LOG   27/10/2014 01:21:18 (TRN nxp pid=20091 sid=49 trid=-1)
[sp.c:sp_send_msg:64]: Message send : type 250, len 0
LOG   27/10/2014 01:21:18 (TRN nxp pid=20091 sid=49 trid=-1)
[sp.c:sp_recv_msg:34]: Message recv : type 210, len 0
LOG   27/10/2014 01:21:18 (TRN nxp pid=20091 sid=49 trid=49)
[sp.c:sp_send_msg:64]: Message send : type 230, len 0
LOG   27/10/2014 01:21:18 (TRN nxp pid=20091 sid=49 trid=49)
[sp.c:sp_recv_msg:34]: Message recv : type 300, len 76
LOG   27/10/2014 01:21:18 (TRN nxp pid=20091 sid=49 trid=49)
[tr_functions.cpp:on_user_statement_begin:102]: User query:
---   DROP DOCUMENT "TreeExport.xml" IN COLLECTION "treeStructures/released"
ERROR 27/10/2014 01:21:20 (TRN nxp pid=20091 sid=49 trid=49)
[btree.cpp:bt_delete_tmpl:273]: (SE1008) Sedna index subsystem error.
Details: Cannot delete object which is not in the btree
LOG   27/10/2014 01:21:20 (TRN nxp pid=20091 sid=49 trid=-1)
[tr_common_funcs.cpp:on_transaction_end:416]: Transaction has been ROLLED
BACK
LOG   27/10/2014 01:21:20 (TRN nxp pid=20091 sid=49 trid=-1)
[sp.c:sp_send_msg:64]: Message send : type 100, len 122
LOG   27/10/2014 01:21:25 (TRN nxp pid=20091 sid=49 trid=-1)
[sp.c:sp_recv_msg:34]: Message recv : type 500, len 0
LOG   27/10/2014 01:21:25 (TRN nxp pid=20091 sid=49 trid=-1)
[sp.c:sp_send_msg:64]: Message send : type 510, len 0
INFO  27/10/2014 01:21:25 (SM nxp pid=19377)
[bm_core.cpp:bm_log_out_io_statistics:537]: IO block reads:0, writes:0
LOG   27/10/2014 01:21:25 (TRN nxp pid=20091 sid=49 trid=-1)
[tr.cpp:TRmain:456]: Session is closed

It's persistent, I could not delete the document at all until I deleted the
indexes on that collection:
  <index name="category_href" backend="btree" as_type="xs:string"
on_path="descendant::element(Category)"
by_path="attribute::attribute(href)" object_name="treeStructures/released"
object_type="collection"/>
  <index name="category_id" backend="btree" as_type="xs:string"
on_path="descendant::element(Category)"
by_path="attribute::attribute(Identifier)"
object_name="treeStructures/released" object_type="collection"/>

Any ideas?

Best regards,
Ivan Lagunov


On Fri, Oct 17, 2014 at 10:05 AM, Ilya Taranov <ep...@so...> wrote:

> Hi, here is the patch over the latest label that should add protocol
> minimal debugging.
>
> On Wed, Oct 15, 2014 at 9:53 AM, Ivan Lagunov <la...@gm...> wrote:
>
>> Hi Ilya,
>>
>> As it's stated in the email topic, I use Sedna 3.5.615. It was built
>> from sources with RelWithDebugInfo flag enabled on the RHEL linux
>> server. If you want to add some additional debug info, I'd suggest adding
>> it on top of the latest night build though. I'm planning to upgrade anyway
>> as there were quite some nice features implemented.
>>
>> @Charles, okay, nice to know this. If only I could provide a consistently
>> failing test case. Currently the issue is quite mysterious.
>>
>> Best regards,
>> Ivan Lagunov
>>
>> On Wed, Oct 15, 2014 at 11:02 AM, Ivan Shcheklein <shc...@gm...>
>> wrote:
>>
>>> Ilya,
>>>
>>> As far as I remember. It is almost the latest dev sources built with
>>> relWithDebInfo. Internal data structures should be compatible.
>>>
>>> Ivan
>>>
>>> On Wed, Oct 15, 2014 at 2:08 AM, Ilya Taranov <ep...@so...>
>>> wrote:
>>>
>>>> I'm not sure, latest build will fix the problem.
>>>>
>>>> It's definitely some client-to-app communication problem.
>>>> If I make a git patch for you with additional debug info, will you be
>>>> able to apply it (assuming, the data file version will be the same)? What
>>>> build do you currently use?
>>>>
>>>> On Sat, Oct 11, 2014 at 12:48 AM, Ivan Lagunov <la...@gm...>
>>>> wrote:
>>>>
>>>>> Hi Ilya,
>>>>>
>>>>> I've analyzed logs for other issue occurrences and it has appeared
>>>>> more complicated.
>>>>>
>>>>> First, could you please clarify about "Session is ready" statement,
>>>>> when exactly is it dumped to the log file? I'm wondering because when the
>>>>> database is hanging, those log entries still appear at some random moments
>>>>> of time.
>>>>>
>>>>> Second, I'm going to provide the investigation details for several
>>>>> more issue occurrences here. Below "fast XQuery" means that it uses index
>>>>> and is expected to be very fast.
>>>>>
>>>>> https://www.dropbox.com/s/2luq53cstc4t3tl/sedna_debug_info1.zip?dl=0
>>>>>
>>>>> se_trn 27173 - DROP DOCUMENT operation, COMMITED
>>>>> se_gov 29197
>>>>> se_sm 29224
>>>>> se_trn 29617 - fast XQuery basictypes:getBasicType, not COMMITED, gdb
>>>>> thread 1 in select
>>>>> se_trn 29820 - Session is ready
>>>>> se_trn 29976 - Session is ready
>>>>>
>>>>> https://www.dropbox.com/s/scmszq9yphyv6xf/sedna_debug_info2.zip?dl=0
>>>>>
>>>>> se_gov 15890
>>>>> se_sm 15899
>>>>> se_trn 21611 - doc-available XQuery, not COMMITED, gbd thread 1 in
>>>>> nanosleep
>>>>> se_trn 21709 - doc('$version') XQuery, COMMITED, gdb thread 1 in select
>>>>> se_trn 21713 - doc('$version') XQuery, COMMITED, gdb thread 1 in select
>>>>> se_trn 21716 - doc('$version') XQuery, COMMITED, gdb thread 1 in select
>>>>> se_trn 21719 - doc('$version') XQuery, COMMITED, gdb thread 1 in select
>>>>> se_trn 21727 - LOAD STDIN operation, not COMMITED, gbd thread 1 in
>>>>> nanosleep
>>>>> se_trn 21996 - fast XQuery basictypes:getBasicType, not COMMITED, gdb
>>>>> thread 1 in select
>>>>> se_trn 22002 - document("$collections") XQuery, not COMMITED, gdb
>>>>> thread 1 in nanosleep
>>>>> se_trn 22010 - Session is ready
>>>>> se_trn 22013 - Session is ready
>>>>> se_trn 22152 - Session is ready
>>>>> se_trn 22295 - Session is ready
>>>>> se_trn 23615 - Session is ready
>>>>> se_trn 24315 - Session is ready
>>>>>
>>>>> Questions:
>>>>> 1. Does nanosleep state mean that the operation is waiting for another
>>>>> one (here it's probably 21996 XQuery in select)?
>>>>> 2. Why does doc-available wait for any other operation? I'd expect it
>>>>> to be non-blocking and to run immediately on the latest data snapshot.
>>>>> 3. Why does doc('$version') wait for any other operation? It's a
>>>>> database metadata being fetched, so I'd also expect it to run immediately.
>>>>>
>>>>> https://www.dropbox.com/s/3ibo4dvzapfdirw/sedna_debug_info3.zip?dl=0
>>>>>
>>>>> se_trn 641 - fast XQuery packages:getPackage, not COMMITED, gdb thread
>>>>> 1 in select
>>>>> se_trn 645 - Session is ready & COMMITED without operation
>>>>> se_trn 648 - Session is ready
>>>>> se_trn 823 - Session is ready
>>>>> se_trn 898 - Session is ready
>>>>> se_gov 8444
>>>>> se_sm 8453
>>>>>
>>>>> Questions:
>>>>> What is going on with 645? There is "Session in ready" statement
>>>>> immediately followed by "COMMITED" statement in the event.log. What
>>>>> operation was committed? Is it possible the order of statements in the
>>>>> event.log is inconsistent? Even so there is no more statements with
>>>>> pid=645, so no operation to be seen that could be committed. Very
>>>>> suspicious.
>>>>>
>>>>> https://www.dropbox.com/s/38idxmpzjv6rvxf/sedna_debug_info4.zip?dl=0
>>>>>
>>>>> se_gov 1532
>>>>> se_sm 1711
>>>>> se_trn 6837 - DROP DOCUMENT operation, not COMMITED, gdb thread 1 in
>>>>> nanosleep
>>>>> se_trn 11703 - LOAD STDIN operation, not COMMITED, gdb thread 1 in
>>>>> nanosleep
>>>>> se_trn 11944 - document("$collections") XQuery, not COMMITED, gdb
>>>>> thread 1 in nanosleep
>>>>> se_trn 11972 - document("$collections") XQuery, not COMMITED, gdb
>>>>> thread 1 in nanosleep
>>>>> se_trn 12139 - XQuery collection("groupingDefinitions/wip"), COMMITED,
>>>>> gdb thread 1 in select
>>>>> se_trn 12183 - fast XQuery basictypes:getBasicType, not COMMITED, gdb
>>>>> thread 1 in select
>>>>> se_trn 12186 - XQuery collection("groupingDefinitions/wip"), COMMITED,
>>>>> gdb thread 1 in select
>>>>> se_trn 12195 - XQuery collection("groupingDefinitions/wip"), COMMITED,
>>>>> gdb thread 1 in select
>>>>> se_trn 12222 - document("$collections") XQuery, not COMMITED, gdb
>>>>> thread 1 in nanosleep
>>>>> se_trn 12302 - fast XQuery documents:getFilesByBasicTypeId, not
>>>>> COMMITED, gdb thread 1 in nanosleep
>>>>> se_trn 12307 - document("$documents") XQuery, not COMMITED, gdb thread
>>>>> 1 in nanosleep
>>>>> se_trn 12310 - document("$collections") XQuery, not COMMITED, gdb
>>>>> thread 1 in nanosleep
>>>>> se_trn 12377 - Session is ready
>>>>> se_trn 12382 - Session is ready
>>>>> se_trn 12423 - Session is ready
>>>>> se_trn 12848 - Session is ready
>>>>>
>>>>> Comment:
>>>>> I understand many of those XQueries are not optimal, they are part of
>>>>> an application developed by other devs. Now I'll take care of improving
>>>>> those operations to replace document("$documents") and
>>>>> document("$collections") with doc-available and col-available accordingly
>>>>> and to use an index for collection("groupingDefinitions/wip").
>>>>>
>>>>> Overall conclusion:
>>>>> There is no single operation/XQuery causing the database hanging.
>>>>> There were at least three different XQueries observed. Taking into account
>>>>> the seemingly random occurrences, it looks like the protocol issue indeed.
>>>>> I don't see how to apply tcpdump in this case as it's Production
>>>>> environment. Any ideas?
>>>>>
>>>>> I'd try to rebuild the database from sources and, perhaps, to take the
>>>>> latest night 3.5 (or 3.6?) build for this. Do you think it makes sense to
>>>>> try?
>>>>>
>>>>> Best regards,
>>>>> Ivan Lagunov
>>>>>
>>>>> On Fri, Oct 10, 2014 at 1:23 AM, Ilya Taranov <ep...@so...>
>>>>> wrote:
>>>>>
>>>>>> Hi, from the top of my head...
>>>>>>
>>>>>> 1. Queries are evaluated in a lazy fashion. Which means, that before
>>>>>> client asks to fetch the next row, nothing is evaluated.
>>>>>>
>>>>>> So select() is a socket read, which is waiting for client to make a
>>>>>> decision on whether to commit (or end RO transaction) or ask for next part
>>>>>> of the result. All locks are only released when end transaction happens.
>>>>>> That's why others are waiting for one single client.
>>>>>>
>>>>>> To my mind, it's a wrong approach, and that's not entirely how I
>>>>>> would do it right now, But that days we had ACID as a primary goal in minds.
>>>>>>
>>>>>> 2. Session is ready, means that there is no ongoing transaction, but
>>>>>> the connection is established.
>>>>>>
>>>>>> 3. No idea in this particular case, but there can be a situation when
>>>>>> session is started without client connection. Recovery/checkpoint, maybe
>>>>>> something else, I'm not sure.
>>>>>>
>>>>>> 4. That dfntly looks strange, but should not cause a lock. Once
>>>>>> transaction is committed, it should release all the locks. So even should
>>>>>> there be orphan sessions, they would not interfere the normal operation.
>>>>>>
>>>>>> 5. Yes that pretty much looks like a problem. Based on logs, it looks
>>>>>> like 17338 is waiting for client to fetch a result.
>>>>>>
>>>>>> That may be a protocol issue, by the way, when as a result of
>>>>>> malformed message both client and sedna are waiting for each other.
>>>>>>
>>>>>> I would guess, that's always happens when
>>>>>> "categorytree:getCategoryTree()" is called, could you please try that? If
>>>>>> it confirms, could you please make a wireshark/tcpdump dump of this
>>>>>> communication?
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 9, 2014 at 6:42 AM, Ivan Lagunov <la...@gm...>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ilya,
>>>>>>>
>>>>>>> Please check my investigations below.
>>>>>>>
>>>>>>> I've collected all the debug information for another hanging today,
>>>>>>> it's available here:
>>>>>>> https://www.dropbox.com/s/g14aer9821fzx05/sedna_debug_info6.zip?dl=0
>>>>>>> Here are the statuses of all Sedna processes:
>>>>>>>
>>>>>>> se_trn 13436 - DROP DOCUMENT operation, COMMITED
>>>>>>> se_trn 16245 - LOAD STDIN operation, COMMITED, no ESTABLISHED
>>>>>>> connection on port 5050 found (only 5151)
>>>>>>> se_trn 17335 - doc('$version') XQuery, COMMITED, gdb thread 1 in
>>>>>>> select () from /lib64/libc.so.6
>>>>>>> se_trn 17338 - XQuery running about 10sec, not COMMITED, gdb thread
>>>>>>> 1 in select () from /lib64/libc.so.6
>>>>>>> se_trn 17410 - XQuery running <1sec, COMMITED
>>>>>>> se_trn 17462 - Session is ready
>>>>>>> se_trn 17554 - Session is ready
>>>>>>> se_trn 18892 - Session is ready
>>>>>>> se_trn 19124 - Session is ready
>>>>>>> se_gov 20441 - gdb threads 1 and 2 in select () from /lib64/libc.so.6
>>>>>>> se_sm 20449
>>>>>>>
>>>>>>> Comments:
>>>>>>> 1. Where gdb is not stated, all threads are in semop () from
>>>>>>> /lib64/libc.so.6 except the last thread that is in pthread_cond_timedwait@@GLIBC_2.3.2
>>>>>>> (). I believe semop is a good state, correct me if I'm wrong.
>>>>>>> 2. All se_trn with "Session is ready" have no other entries in the
>>>>>>> latest event.log. As I collect only the latest event.log, I cannot check
>>>>>>> what was the last operation.
>>>>>>> 3. All se_trn processes have two ESTABLISHED connections in netstat
>>>>>>> for ports 5050 and 5151 if not stated otherwise.
>>>>>>> 4. se_trn 17338, 17410, 17462 were created by the same application
>>>>>>> that is under suspicion. The database unfreezed after the restart of that
>>>>>>> application.
>>>>>>>
>>>>>>> Issues/questions:
>>>>>>> 1. What exactly does it mean when a thread is in select()? It seems
>>>>>>> the database is waiting but for the client or for some internal operation,
>>>>>>> e.g. fetching the data. Please clarify.
>>>>>>> 2. What exactly does "Session is ready" mean? When does it appear in
>>>>>>> the event.log? I believe those connections are okay but I'm curious to know.
>>>>>>> 3. se_trn 16245 - how is it possible that there is no ESTABLISHED
>>>>>>> connection on port 5050 while the process is still alive? An issue?
>>>>>>> 4. se_trn 17335 - how is it possible that thread is in select()
>>>>>>> while the transaction has been COMMITED in event.log? An issue?
>>>>>>> 5. se_trn 17338 - looks like a root cause of Sedna hanging.
>>>>>>> Currently I run it via XML:DB API implementation mentioned earlier, more
>>>>>>> specifically I run the XQuery via XQueryService.query("xquery string")
>>>>>>> call. Besides, I always try to close the connection in the Java finally
>>>>>>> block, so it looks roughly like this:
>>>>>>>
>>>>>>> try {
>>>>>>>     rootCollection = DatabaseManager.getCollection(...);
>>>>>>>     ... do something ...
>>>>>>> } finally {
>>>>>>>     rootCollection.close();
>>>>>>> }
>>>>>>>
>>>>>>> According to event.log the XQuery was started at 09/10/2014 12:00:01.
>>>>>>> In the application log somewhere between 12:01:53 and 12:05:00
>>>>>>> appeared a message:
>>>>>>>
>>>>>>>> Warning: an open Sedna XML:DB Collection/Connection was just
>>>>>>>> automatically closed due to inactivity, you SHOULD close these manually.
>>>>>>>
>>>>>>> It seems to mean that the connection was considered as abandoned and
>>>>>>> removed by XML:DB API implementation (see above conn-log-abandoned=true
>>>>>>> and conn-remove-abandoned-timeout=120000).
>>>>>>> For me it looks like the client waited for 2 minutes but did not get
>>>>>>> any response from the database. So there was an attempt to close the
>>>>>>> connection. It's easy to understand as the database hanged at about
>>>>>>> 12:00:02 when the last transaction was COMMITED. So it may be even that the
>>>>>>> root cause is hidden somewhere deeper. For example, this long XQuery could
>>>>>>> cause a deadlock.
>>>>>>>
>>>>>>> I understand it may be an issue in XML:DB API implementation but its
>>>>>>> support was stopped by Charles Foster. Unfortunately, we cannot afford to
>>>>>>> migrate to XQJ API implementation yet. The most tricky part to understand
>>>>>>> is why we had not faced similar issues before recent occurrence when it has
>>>>>>> become a regular major issue immediately. There were no server updates and
>>>>>>> no changes that could affect the way transactions/connections are handled.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ivan Lagunov
>>>>>>>
>>>>>>> On Thu, Oct 9, 2014 at 1:33 PM, Ivan Lagunov <la...@gm...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry, copied the wrong link for the connection pool documentation.
>>>>>>>> Here is the correct one:
>>>>>>>> http://www.cfoster.net/pdf/sedna/xmldb/documentation.pdf#page=10
>>>>>>>>
>>>>>>>> Besides, I use the following values for the connection pool
>>>>>>>> parameters in the suspected application:
>>>>>>>> conn-max-active=50
>>>>>>>> conn-max-wait=30000
>>>>>>>> conn-log-abandoned=true
>>>>>>>> conn-remove-abandoned-timeout=120000
>>>>>>>> conn-abandoned-check-interval=5000
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Ivan Lagunov
>>>>>>>>
>>>>>>>> On Thu, Oct 9, 2014 at 1:28 PM, Ivan Lagunov <la...@gm...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Ilya,
>>>>>>>>>
>>>>>>>>> I use Charles Foster implementation of XML:DB API, version 1.2.6
>>>>>>>>> without any customizations:
>>>>>>>>> http://www.cfoster.net/sedna/xmldb/download/
>>>>>>>>>
>>>>>>>>> Here is the link again to the connection pool documentation (it's
>>>>>>>>> a part of the above implementation):
>>>>>>>>> http://www.cfoster.net/sedna/xmldb/download/
>>>>>>>>>
>>>>>>>>> Meanwhile, I've identified the application that makes Sedna wait.
>>>>>>>>> At least restarting the application has unfreezed the database twice
>>>>>>>>> already. I'll analyze the application log along with Sedna event.log
>>>>>>>>> shortly. Hopefully, I'll find the root cause.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Ivan Lagunov
>>>>>>>>>
>>>>>>>>> On Thu, Oct 9, 2014 at 3:50 AM, Ilya Taranov <ep...@so...
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I can easily see the scenario, where one of the clients doesn't
>>>>>>>>>> terminate the transaction properly. Unfortunately, current application
>>>>>>>>>> timeout doesn't apply for client hangs. The issue can be easily fixed,
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Unfortunately, even fix still would not solve your problem, you
>>>>>>>>>> would still have application timeout wait on locks. Probably the most
>>>>>>>>>> straightforward way to find the cause for you would be to log connection
>>>>>>>>>> checkins/checkouts at the client driver level.
>>>>>>>>>>
>>>>>>>>>> Could you please give me a link to the connection pool + driver
>>>>>>>>>> you currently use (including all your changes if possible) so I could help
>>>>>>>>>> you with that?
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 7, 2014 at 4:37 AM, Ivan Lagunov <la...@gm...>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ilya,
>>>>>>>>>>>
>>>>>>>>>>> Your suggestion seems to be correct. After I've restarted three
>>>>>>>>>>> Java applications that had connections to Sedna, the database has resumed
>>>>>>>>>>> processing operations. Here is the event.log where Sedna has hanged between
>>>>>>>>>>> 12:35:03 and 13:06:05:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://www.dropbox.com/s/pzafuvzcvfchxwa/event-2014-10-07-13-15-04-018.zip?dl=0
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure which of the applications caused it though. It
>>>>>>>>>>> still worries me as the queries before hanging look quite simple and I
>>>>>>>>>>> cannot understand which client operation could have hanged the whole
>>>>>>>>>>> database.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Ivan Lagunov
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 6, 2014 at 12:29 PM, Ivan Lagunov <la...@gm...
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Ilya,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, I use a connection pool via Sedna XML:DB API
>>>>>>>>>>>> implementation. It's documented here:
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.cfoster.net/pdf/sedna/xmldb/documentation.pdf#page=10
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, there is no support for active connection
>>>>>>>>>>>> timeout. This could be the cause if there was a query hanging for some
>>>>>>>>>>>> reason. Still if some query hangs forever (that should not as it's in my
>>>>>>>>>>>> case) it looks more like a database bug/deadlock situation.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll try as you suggest next time.
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. Sorry for the second email as I've forgotten to add CC
>>>>>>>>>>>> first.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Ivan Lagunov
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Oct 3, 2014 at 10:23 PM, Ilya Taranov <
>>>>>>>>>>>> ep...@so...> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Ivan, do you have a connection pool? I see one of the se_trn
>>>>>>>>>>>>> processes waiting for socket read.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you don't have a connection pool, it just might be a client
>>>>>>>>>>>>> problem. I'll take a deeper look anyway, just next time it happens, could
>>>>>>>>>>>>> you please try to shutdown the client thread, if it's even possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Oct 2, 2014 at 5:30 AM, Ivan Lagunov <
>>>>>>>>>>>>> la...@gm...> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been facing major issues with Sedna hanging regularly
>>>>>>>>>>>>>> for the last two weeks (after several months of stable work). I tried to do
>>>>>>>>>>>>>> the complete export-import of data using se_exp but Sedna still hangs
>>>>>>>>>>>>>> regularly. Looking into event.log does not help as queries differ every
>>>>>>>>>>>>>> time. Sometimes it looks like Sedna hangs after some heavy query (with huge
>>>>>>>>>>>>>> response) while in other cases there are no heavy queries to suspect. I've
>>>>>>>>>>>>>> written a script to collect the following logs/dumps you will find in the
>>>>>>>>>>>>>> archives:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> event.log - the last Sedna event log till the moment DB hangs
>>>>>>>>>>>>>> and is stopped;
>>>>>>>>>>>>>> sedna_procs.txt - list of all Sedna processes (ps -ef);
>>>>>>>>>>>>>> netstat.txt - list of all ESTABLISHED connections;
>>>>>>>>>>>>>> ipcs.txt - results of ipcs -s;
>>>>>>>>>>>>>> <pid>.txt - results of gdb with bt for all threads of a Sedna
>>>>>>>>>>>>>> process (available for all processes at the moment of hanging). This is
>>>>>>>>>>>>>> collected automatically for threads 1-4 of each process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm providing two sets of logs/dumps for two separate hanging
>>>>>>>>>>>>>> events:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 25th September - seemed to hang without any heavy queries:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.dropbox.com/s/2luq53cstc4t3tl/sedna_debug_info1.zip?dl=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2nd October - hanged after a heavy query:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.dropbox.com/s/scmszq9yphyv6xf/sedna_debug_info2.zip?dl=0
>>>>>>>>>>>>>> The heavy query occurs at LOG   02/10/2014 11:10:20
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you can find out the cause of the hangings, it'd be the
>>>>>>>>>>>>>> best. Otherwise it'd be nice to hear some recommendations at least on how
>>>>>>>>>>>>>> to avoid these issues/minimize an impact. May be I need to upgrade to some
>>>>>>>>>>>>>> later version but I'm currently using 3.5.615 that is already a night build
>>>>>>>>>>>>>> that we had to upgrade to after some older issue (reported back in
>>>>>>>>>>>>>> March-April of 2013). I'd try some 3.5 or 3.6 version but without knowing
>>>>>>>>>>>>>> the reason there is no guarantee the potential bug is fixed in the later
>>>>>>>>>>>>>> night build.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also it'd be really nice if there was an official release for
>>>>>>>>>>>>>> 3.5 with all the latest bug fixes (I see the last night build is 3.5.627 of
>>>>>>>>>>>>>> 8th May 2013).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Ivan Lagunov
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>>>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog
>>>>>>>>>>>>>> Analyzer
>>>>>>>>>>>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI
>>>>>>>>>>>>>> DSS Reports
>>>>>>>>>>>>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download
>>>>>>>>>>>>>> White paper
>>>>>>>>>>>>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog
>>>>>>>>>>>>>> Analyzer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Sedna-discussion mailing list
>>>>>>>>>>>>>> Sed...@li...
>>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>