[SSI-devel] [ ssic-linux-Bugs-1283892 ] Cluster Hangs on write()

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #1283892, was opened at 2005-09-07 08:30
Message generated for change (Comment added) made by rogertsang
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=405834&aid=1283892&group_id=32541

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Filesystem
Group: v1.9.1
Status: Open
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Cluster Hangs on write()

Initial Comment:
On my 11 node cluster running FC3 and OpenSSi 1.9.1 
the cluster hangs on write() if the file is bigger than 
approx. 300MB. mounting the cfs disk with sync option 
solves the problem, but provides bad performance.
I found another entry in the list telling that with a manual 
sync everthing springs to live again.

----------------------------------------------------------------------

>Comment By: Roger Tsang (rogertsang)
Date: 2007-03-25 03:57

Message:
Logged In: YES 
user_id=1246761
Originator: NO

Seems to be resolved by the patch at
http://article.gmane.org/gmane.linux.cluster.ssic.devel/5253

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-10-30 10:28

Message:
Logged In: NO 

One way to reproduce this is mount a physical partition on a
non-initnode non-chard.  Copy to it a large file twice the
size of that node's vm.dirty_threshold.  Wait until dirty
pages in /proc/meminfo rises above vm.dirty_threshold limit
and for the copy process to hang at blk_congestion_wait.  Do
a sync at that node.  Look up /proc/meminfo on that node,
dirty pages should still be above vm.dirty_threshold.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-10-30 10:04

Message:
Logged In: NO 

It seems the problem stems from pdflush kthread not being
woken on non-initnodes; when the CFS server (CFS mount) is
not on the initnode.  In fact it even looks like the pdflush
process is woken for background_writeout on the initnode,
but should be woken on the CFS server (non-initnode).  So
when writing very large files - bigger than
vm.dirty_threshold of the remote node - a number of
icssvr_daemon's writing to the same file would hang the
cluster waiting for IO in balance_dirty_pages and the CFS
client would be hung waiting for rcfsd_write to complete.  I
can't reproduce this when writing to the CFS server on the
initnode.  There either is a failure at writeback_inodes
call or pdflush_operation call to pdflush's
background_writeout function.

----------------------------------------------------------------------

Comment By: Roger Tsang (rogertsang)
Date: 2005-09-29 01:14

Message:
Logged In: YES 
user_id=1246761

A follow up of this bug is here
http://marc.theaimsgroup.com/?l=ssic-linux-devel&m=112744789721893&w=2

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-09 01:55

Message:
Logged In: NO 

>What is the underlying filesystem and device that you were
>writing to?  Is this a remote CFS write or write to local
>device? 
Usually its a Remote device, a node writes back to the
masternode. It allways happens wenn two nodes write at the
same time.

> Are you writing to device-mapper device?  
If you ask wether the device is a distributed block device,
the answer is no.

>Can you print out the backtrace of the process from kdb on
the node
>that you are writing from?

because it usually happens with parallel processes, I will
see to do that.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-09 01:55

Message:
Logged In: NO 

>What is the underlying filesystem and device that you were
>writing to?  Is this a remote CFS write or write to local
>device? 
Usually its a Remote device, a node writes back to the
masternode. It allways happens wenn two nodes write at the
same time.

> Are you writing to device-mapper device?  
If you ask wether the device is a distributed block device,
the answer is no.

>Can you print out the backtrace of the process from kdb on
the node
>that you are writing from?

because it usually happens with parallel processes, I will
see to do that.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-08 11:04

Message:
Logged In: NO 

>What is the underlying filesystem and device that you were
>writing to?  Is this a remote CFS write or write to local
>device? 
Usually its a Remote device, a node writes back to the
masternode. It allways happens wenn two nodes write at the
same time.

> Are you writing to device-mapper device?  
If you ask wether the device is a distributed block device,
the answer is no.

>Can you print out the backtrace of the process from kdb on
the node
>that you are writing from?

because it usually happens with parallel processes, I will
see to do that.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-08 06:51

Message:
Logged In: NO 

>What is the underlying filesystem and device that you were
>writing to?  Is this a remote CFS write or write to local
>device? 
Usually its a Remote device, a node writes back to the
masternode. It allways happens wenn two nodes write at the
same time.

> Are you writing to device-mapper device?  
If you ask wether the device is a distributed block device,
the answer is no.

>Can you print out the backtrace of the process from kdb on
the node
>that you are writing from?

because it usually happens with parallel processes, I will
see to do that.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-08 06:47

Message:
Logged In: NO 

An occational sync solves the problem and a possible
woarkaround is to do a sync, say every minute or so. But its
not a very convinient way.
Any ideas ? 

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-09-07 17:33

Message:
Logged In: NO 

What is the underlying filesystem and device that you were
writing to?  Is this a remote CFS write or write to local
device?  Are you writing to device-mapper device?  Can you
print out the backtrace of the process from kdb on the node
that you are writing from?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=405834&aid=1283892&group_id=32541