commonsvfsgrid-developers

Re: [Commonsvfsgrid-developers] FW: vfs threading

[Markus B. <ma...@vp...>]

All right, just to let all of you know: I am now using ThreadLocal to 
create one FsManager per thread. That was easy enough to implement, it 
seems to work well and I'm not seeing any unexpected behaviour.

Thanks heaps for your help. That was a really important fix for me...

Cheers,
Markus

Meredith, David (STFC,DL,ESC) wrote:
> For the benefit of the list. 
>
>  
> Hi Dave.
>
> ----- "David Meredith (STFC,DL,ESC)" <dav...@st...> wrote:
>
>   
>> 1) create a new DefaultFileSystemManager instance per thread (note, if
>> using VFSUtil, this requires a new call to createNewFsManager for each
>> thread) e.g.
>>
>> DefaultFileSystemManager fsManager = new DefaultFileSystemManager();
>>
>> fsManager.addProvider("gsiftp", new GridFtpFileProvider());
>>     
>
> I tried that now. To test, I created a new fsManager for every transfer/file access. And that worked without a single glitch. Did 100 uploads, 12 concurrently at each given point in time, and none of them failed. Not sure what the overhead of this solution is, but I definitely prefer the stability to possible speed decreases. Normally, around 15 of those uploads fail.
>
>   
>> 2) create new FileSystemOptions instance per thread and populate with
>> credentials etc .
>>
>> 3) resolve file using the thread local fsManager, e.g.
>>
>> FileObject relativeToFO_ = fsManager.resolveFile(gridftpUri, opts);
>>     
>
> Might try this next week if I find some time.
>
>   
>> Test: here check here that a new GridFtpFileSystem is created on the
>> resolve file operation for each and every thread in
>> GridFtpFileProvider.doCreateFileSystem() .
>>     
>
> That should be easy enough to figure out. Will try that after the weekend...
>
> Cheers,
> Markus
>
>   

[Commonsvfsgrid-developers] FW: vfs threading

[Meredith, D. (STFC,DL,ESC) <dav...@st...>]

For the benefit of the list. 

 
Hi Dave.

----- "David Meredith (STFC,DL,ESC)" <dav...@st...> wrote:

> 1) create a new DefaultFileSystemManager instance per thread (note, if
> using VFSUtil, this requires a new call to createNewFsManager for each
> thread) e.g.
> 
> DefaultFileSystemManager fsManager = new DefaultFileSystemManager();
> 
> fsManager.addProvider("gsiftp", new GridFtpFileProvider());

I tried that now. To test, I created a new fsManager for every transfer/file access. And that worked without a single glitch. Did 100 uploads, 12 concurrently at each given point in time, and none of them failed. Not sure what the overhead of this solution is, but I definitely prefer the stability to possible speed decreases. Normally, around 15 of those uploads fail.

> 
> 2) create new FileSystemOptions instance per thread and populate with
> credentials etc .
> 
> 3) resolve file using the thread local fsManager, e.g.
> 
> FileObject relativeToFO_ = fsManager.resolveFile(gridftpUri, opts);

Might try this next week if I find some time.

> 
> Test: here check here that a new GridFtpFileSystem is created on the
> resolve file operation for each and every thread in
> GridFtpFileProvider.doCreateFileSystem() .

That should be easy enough to figure out. Will try that after the weekend...

Cheers,
Markus

-- 
Scanned by iCritical.

Re: [Commonsvfsgrid-developers] File listing error

[Meredith, D. (STFC,DL,ESC) <dav...@st...>]

hi Markus, 

i have seen a similar error before - it originated because of an invalid file name (started with a \ char) which upset vfs and caused exceptions like below. however, i subsequently added a patch that i thought? addressed this (unless there is a different cause). I'll take a look over the weekend - currently out of office, back next monday. 

dave

------------------------ 
David Meredith 
STFC eScience Centre 
Daresbury Laboratory 
Warrington Cheshire 
WA4 4AD 
Tel: 01925 603762(Direct Line) 
email: dav...@st... 



-----Original Message-----
From: Markus Binsteiner [mailto:ma...@vp...]
Sent: Tue 01/09/2009 11:19 PM
To: com...@li...
Subject: [Commonsvfsgrid-developers] File listing error
 
Hi. Me again....

I'm having another one of those problems I can't reliably reproduce:

I wrote this method which recursively lists (gridftp-)directories in 
order to get a listing of all the children files of a root folder and 
their absolute paths. That works most of the time without a problem. 
But, approximately once in ten times it fails with an error that looks like:

org.apache.commons.vfs.FileSystemException: Could not list the contents 
of folder "{0}".
    at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpMlsxFileObject.statChildren(GridFtpMlsxFileObject.java:376)
    at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpMlsxFileObject.doListChildrenResolved(GridFtpMlsxFileObject.java:280)
    at 
org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:589)
    at 
org.vpac.grisu.control.serviceInterfaces.AbstractServiceInterface.getFolderListing(AbstractServiceInterface.java:1749)
    at 
org.vpac.grisu.control.serviceInterfaces.AbstractServiceInterface.ls(AbstractServiceInterface.java:1820)
    at 
org.vpac.grisu.frontend.model.job.JobObject.listJobDirectory(JobObject.java:708)
    at 
org.vpac.grisu.frontend.model.job.MultiPartJobObject.downloadResults(MultiPartJobObject.java:673)
    at 
org.vpac.grisu.clients.blender.GridBlenderCheck.execute(GridBlenderCheck.java:140)
    at 
org.vpac.grisu.clients.blender.GrisuBlender.main(GrisuBlender.java:41)
Caused by: org.globus.ftp.exception.ServerException: Server refused 
performing the request. Custom message:  (error code 1) [Nested 
exception message:  Custom message: Unexpected reply: 451 active 
connection to server failed
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:520)
    at java.net.Socket.connect(Socket.java:470)
    at java.net.Socket.<init>(Socket.java:367)
    at java.net.Socket.<init>(Socket.java:267)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:74)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:53)
    at 
org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:70)
    at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71)
    at java.lang.Thread.run(Thread.java:595)
] [Nested exception is 
org.globus.ftp.exception.UnexpectedReplyCodeException:  Custom message: 
Unexpected reply: 451 active connection to server failed
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:520)
    at java.net.Socket.connect(Socket.java:470)
    at java.net.Socket.<init>(Socket.java:367)
    at java.net.Socket.<init>(Socket.java:267)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:74)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:53)
    at 
org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:70)
    at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71)
    at java.lang.Thread.run(Thread.java:595)
]
    at 
org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:101)
    at 
org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:110)
    at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
    at java.lang.Thread.run(Thread.java:595)


Did anybody see similar errors? Any ideas what could be the problem?
I didn't do any real investigation into this, but I will soon. Just 
wanted to know whether anybody (well, Dave :-) )has got an idea where to 
start looking, or maybe a hunch...

By the way: this all happens in the main thread, in case you are 
wondering. No fancy threading involved...

Cheers,
Markus

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers

-- 
Scanned by iCritical.

[Commonsvfsgrid-developers] File listing error

[Markus B. <ma...@vp...>]

Hi. Me again....

I'm having another one of those problems I can't reliably reproduce:

I wrote this method which recursively lists (gridftp-)directories in 
order to get a listing of all the children files of a root folder and 
their absolute paths. That works most of the time without a problem. 
But, approximately once in ten times it fails with an error that looks like:

org.apache.commons.vfs.FileSystemException: Could not list the contents 
of folder "{0}".
    at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpMlsxFileObject.statChildren(GridFtpMlsxFileObject.java:376)
    at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpMlsxFileObject.doListChildrenResolved(GridFtpMlsxFileObject.java:280)
    at 
org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:589)
    at 
org.vpac.grisu.control.serviceInterfaces.AbstractServiceInterface.getFolderListing(AbstractServiceInterface.java:1749)
    at 
org.vpac.grisu.control.serviceInterfaces.AbstractServiceInterface.ls(AbstractServiceInterface.java:1820)
    at 
org.vpac.grisu.frontend.model.job.JobObject.listJobDirectory(JobObject.java:708)
    at 
org.vpac.grisu.frontend.model.job.MultiPartJobObject.downloadResults(MultiPartJobObject.java:673)
    at 
org.vpac.grisu.clients.blender.GridBlenderCheck.execute(GridBlenderCheck.java:140)
    at 
org.vpac.grisu.clients.blender.GrisuBlender.main(GrisuBlender.java:41)
Caused by: org.globus.ftp.exception.ServerException: Server refused 
performing the request. Custom message:  (error code 1) [Nested 
exception message:  Custom message: Unexpected reply: 451 active 
connection to server failed
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:520)
    at java.net.Socket.connect(Socket.java:470)
    at java.net.Socket.<init>(Socket.java:367)
    at java.net.Socket.<init>(Socket.java:267)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:74)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:53)
    at 
org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:70)
    at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71)
    at java.lang.Thread.run(Thread.java:595)
] [Nested exception is 
org.globus.ftp.exception.UnexpectedReplyCodeException:  Custom message: 
Unexpected reply: 451 active connection to server failed
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:520)
    at java.net.Socket.connect(Socket.java:470)
    at java.net.Socket.<init>(Socket.java:367)
    at java.net.Socket.<init>(Socket.java:267)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:74)
    at org.globus.net.SocketFactory.createSocket(SocketFactory.java:53)
    at 
org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:70)
    at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71)
    at java.lang.Thread.run(Thread.java:595)
]
    at 
org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:101)
    at 
org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:110)
    at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
    at java.lang.Thread.run(Thread.java:595)


Did anybody see similar errors? Any ideas what could be the problem?
I didn't do any real investigation into this, but I will soon. Just 
wanted to know whether anybody (well, Dave :-) )has got an idea where to 
start looking, or maybe a hunch...

By the way: this all happens in the main thread, in case you are 
wondering. No fancy threading involved...

Cheers,
Markus

Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Meredith, D. (STFC,DL,ESC) <dav...@st...>]

hi folks, 
sorry, currently in an ICAT meeting for next two days - i will get back to you as soon as i can (prob next week now). 


------------------------ 
David Meredith 
STFC eScience Centre 
Daresbury Laboratory 
Warrington Cheshire 
WA4 4AD 
Tel: 01925 603762(Direct Line) 
email: dav...@st... 



-----Original Message-----
From: Markus Binsteiner [mailto:ma...@vp...]
Sent: Sun 23/08/2009 7:49 AM
To: commonsvfsgrid-developers
Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads
 
Hi.

Did some more investigating...

I'm pretty sure now that the problem only occurs when I access the *same* filesystem/gridftp-server concurrently.

If I have 10 threads accessing different filesystems everything is all right.

So, maybe it's a limitation in the cog gridftp client class? It's supposed to support parallel transfers, but those probably need to be setup differently than what I'm doing. 
To test this I tried to change the commons-vfs-grid code to create a new GridFTPClient object for every filesystem access. But I'm not sure whether I really succeeded doing that or whether it just didn't help...

Any other suggestions?

Cheers,
Markus



----- "David Meredith (STFC,DL,ESC)" <dav...@st...> wrote:

> Hi Markus, 
> 
> Take a look at getClient() and createAndConfigureGridFtpClient() in
> GridFtpFileSystem.java. Note line 159: idleClient.changeDir(defdir);
> I'll have a crack at this later today also. 
> 
> I didn't really get very far with the many small files issue (again
> due to time issues). We also have to update to jglobus 1.7 which may
> help. I suspect that using the gridftp get and put (rather than byte
> stream) to get and put between file:// and gsiftp:// will also address
> this. 
> 
> dave 
> 
> 
> -----Original Message-----
> From: Markus Binsteiner [mailto:ma...@vp...] 
> Sent: 18 August 2009 09:26
> To: Meredith, David (STFC,DL,ESC)
> Cc: com...@li...
> Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when using
> multiple threads
> 
> Hi Dave.
> 
> Thanks for looking into that. Inline...
> 
> ----- "David Meredith (STFC,DL,ESC)" <dav...@st...>
> wrote:
> 
> > Hi Markus, 
> > 
> > I was thinking about the issues you reported last-night and I may
> > have
> > an idea why its failing. I think it may? be related to a feature
> that
> > I
> > added to the GridFTPFileSystem that prevented gridftp servers from
> > un-mounting the home directory given prolonged idle time delays
> (i.e.
> > the tcp connection stayed alive but after say 60secs idle time,
> some
> > of
> > the NGS gridftp servers would un-mount the users home dir which
> > caused
> > any subsequent VFS operations to fail). Adding this feature to punt
> > the
> > remote server to mount the users home dir if un-mounted sounds like
> a
> > potential candidate for the multi-threading issues. 
> 
> Would it be easy enough for me to try to disable this feature so I can
> test whether I get more reliabe multi-threaded transfers? Where would
> I have to look?
> 
> > The performance issue is (I think) due to the following: 
> > Presumably you are using the vfs copy function. If true, then this
> > function only uses optimisations when copying between two gridftp
> > servers for doing the 3rd party transfer. Otherwise, the method
> falls
> > back to the VFS byte streaming approach, e.g. when copying between
> > two
> > different file systems, which includes local file:// and gsiftp://.
> > This default byte streaming approach does not use any other
> protocol
> > specific optimisations to improve performance, rather it simply
> > streams
> > bytes using a input and output streams (bit pipe). Therefore, some
> > work
> > is still needed to improve this method, specifically for adding
> > switches
> > to use any optimisations for copying between local file:// and a
> > remote
> > protocol (e.g. gsiftp get and put). 
> 
> At this stage, I'm not overly concerned about the performance issues.
> At the moment I'm just piping together input & output streams myself
> (input stream from a datahandler that gets it's data from a webservice
> method) and that's probably a much bigger bottleneck than anything
> else.
> Also, I'm not really transferring that many or really big files, so I
> can live with that.
> 
> > 
> > These issues definitely need looking at, but unfortunately I am on
> > leave
> > and out of  office a great deal over the next couple of weeks
> > (however,
> > I'll take a look when I can and keep you updated regarding
> progress).
> > If
> > a proposal that I recently submitted gets funded, this will give me
> > time
> > to do all this, fingers crossed, otherwise its best effort). 
> 
> That would be great. Good luck. 
> ...to all of us :-)
> 
> BTW, anything new regarding the transfer of lots-of-small files
> issue?
> 
> Cheers,
> Markus
> 
> 
> > 
> >  
> > dave
> > 
> > 
> > 
> > 
> > -----Original Message-----
> > From: Markus Binsteiner [mailto:ma...@vp...] 
> > Sent: 17 August 2009 04:07
> > To: com...@li...
> > Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when
> using
> > multiple threads
> > 
> > Some more info:
> > 
> > First, I forgot to say: if I use only a single thread, nothing ever
> > fails.
> > 
> > Also, we did some more testing and captured some gridftp server
> logs.
> > 
> > For a connection that works, this is what the logs say:
> > 
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > PASV^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 227 Entering Passive Mode (132,181,39,23,156,66)^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > TYPE A^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 200 Type set to A.^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > CWD /home/users/bestgrid/grid-admin^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 250 CWD command successful.^M
> > [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> >
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> > .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> > C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > 250 End.^M
> > [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368:
> > [CLIENT]:
> > PWD^M
> > [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 257 "/home/users/bestgrid/grid-admin" is current directory.^M
> > 
> > A similar transaction (I think) failing:
> > 
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > PASV^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 227 Entering Passive Mode (132,181,39,23,156,64)^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > TYPE A^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 200 Type set to A.^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > CWD /home/users/bestgrid/grid-admin^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 250 CWD command successful.^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> >
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> > .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> > C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > 250 End.^M
> > [31181] Mon Aug 17 14:33:43 2009 :: Closed connection from 
> > hqrouter.vpac.org:40367
> > 
> > So, it seems somehow the client can't interpret the server response
> > for 
> > the PWD or, more likely, the server doesn't even get the request
> > (since 
> > it doesn't say: [CLIENT]: PWD, but just closes the connection).
> > 
> > In the client debug log I have an exception like:
> > 
> > ERROR - isConnected error
> > org.globus.ftp.exception.ServerException: The server uses unknown 
> > communication protool. Custom message: (error code 2) [Nested
> > exception 
> > message: Custom message: Cannot parse 'PWD' reply: 200 Type set to
> > I.]. 
> > Nested exception is org.globus.ftp.exception.FTPReplyParseException:
> 
> > Custom message: Cannot parse 'PWD' reply: 200 Type set to I.
> > at org.globus.ftp.FTPClient.getCurrentDir(FTPClient.java:345)
> > at 
> >
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.isC
> > onnected(GridFtpFileSystem.java:176)
> > at 
> >
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.get
> > Client(GridFtpFileSystem.java:144)
> > 
> > This seems suspicious: Cannot parse 'PWD' reply: 200 Type set to I.
> > 
> > Any chance the client is mixing up connections somehow?
> > 
> > Any other ideas?
> > 
> > Cheers,
> > Markus
> > 
> > Markus Binsteiner wrote:
> > > Hi.
> > >
> > > I'm wondering whether anybody here has got experience with this:
> > >
> > > I wrote a grid job submission client that tries to make job
> > submission
> > 
> > > as easy as possible for the user. For example, it uploads files
> > from
> > the 
> > > users desktop to the jobdirectory on the cluster where the job
> will
> > run. 
> > > I'm using commons-vfs-grid to do all the gridftp things.
> > >
> > > Now, every now and then the upload of one of the input files
> fails.
> > The 
> > > reason is not always the same, sometimes its for example:
> > > Could not determine the type of file
> > "gsiftp://ng2.vpac.org/home/grid-admin"
> > > , sometimes it's:
> > > Could not write to 
> > >
> >
> "gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus
> > _Binsteiner/simpleTestTarget.txt82
> > >
> > > Sometimes it's something else...
> > >
> > > If I repeat the  same jobsubmission a few seconds later it works
> > fine.
> > >
> > > I tried to understand what's happening but it's really hard
> because
> > I 
> > > can't reproduce the errors, they just happen once in a while.
> > >
> > > To get more data, I wrote a gridftp test client which executes
> > several
> > 
> > > different tests, and one thing I can reproduce is that, if I use
> > more 
> > > threads that do the same action (e.g. upload several files at
> once
> > to 
> > > the same gridftp location), the more threads I use, the more
> errors
> > I 
> > > get. That may have nothing to do with my initial error, but it's
> > still
> > a 
> > > problem since I need to do multiple transfers in different threads
> 
> > > sometimes.
> > >
> > > Roughly, if I use 10 threads, I get about 20 failures out of 100
> > file 
> > > transmissions. Sometimes more, sometimes less. For 3 threads, I
> get
> > 
> > > around 7.
> > >
> > > I use a new FileSystem for every transfer. I set it up something
> > like:
> > >
> > > -----------------
> > >             fsmanager = VFSUtil.createNewFsManager(false, false,
> > true,
> > true,
> > >                     true, true, true, null);
> > >             // the fsmanager is created only once
> > > -----------------
> > >
> > >         FileSystemOptions opts = new FileSystemOptions();
> > >
> > >         if (fileUrl.startsWith("gsiftp")) {
> > >             GridFtpFileSystemConfigBuilder builder = 
> > > GridFtpFileSystemConfigBuilder
> > >                     .getInstance();
> > >             builder.setGSSCredential(opts, gsscredential);
> > >         }
> > >
> > >         FileObject fileRoot =
> > fsmanager.resolveFile(mp.getRootUrl(),
> > opts);
> > >
> > >         FileSystem fileBase = fileRoot.getFileSystem();
> > >
> > >        // do the transfer....
> > >
> > > Anything wrong here?
> > >
> > > This not only happens if I do simultaneous upload. It also
> happens
> > with 
> > > download or if I just do an ls on a remote directory
> > simultaneously...
> > >
> > > One theory I have is that maybe somehow the client times out more
> > easily 
> > > when there are several connections at the same time, but I
> wouldn't
> > have 
> > > a clue as to where to look for something like this.
> > >
> > > I'm not sure whether commons-vfs is threadsafe, but since I'm
> > creating
> > a 
> > > new Filesystem for every transfer that shouldn't really be an
> > issue,
> > or 
> > > am I wrong? Or is there some automatic pooling or such happening
> > down 
> > > the track either in the commons-vfs or cog libraries?
> > >
> > > Any ideas on what could be the issue? I don't think it's a server
> > error 
> > > because I used the commandline c client to do something
> equivalent
> > and
> > 
> > > that was fine (and much much faster :-) )...
> > >
> > > Cheers,
> > > Markus
> > >
> > >
> > >
> > >
> >
> ------------------------------------------------------------------------
> > ------
> > > Let Crystal Reports handle the reporting - Free Crystal Reports
> > 2008
> > 30-Day 
> > > trial. Simplify your report design, integration and deployment -
> > and
> > focus on 
> > > what you do best, core application coding. Discover what's new
> with
> > 
> > > Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> > > _______________________________________________
> > > Commonsvfsgrid-developers mailing list
> > > Com...@li...
> > >
> >
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> > >   
> > 
> > 
> >
> ------------------------------------------------------------------------
> > ------
> > Let Crystal Reports handle the reporting - Free Crystal Reports
> 2008
> > 30-Day 
> > trial. Simplify your report design, integration and deployment -
> and
> > focus on 
> > what you do best, core application coding. Discover what's new with
> 
> > Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> > _______________________________________________
> > Commonsvfsgrid-developers mailing list
> > Com...@li...
> >
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> > --
> > Scanned by iCritical.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers

-- 
Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Markus B. <ma...@vp...>]

Hi.

Did some more investigating...

I'm pretty sure now that the problem only occurs when I access the *same* filesystem/gridftp-server concurrently.

If I have 10 threads accessing different filesystems everything is all right.

So, maybe it's a limitation in the cog gridftp client class? It's supposed to support parallel transfers, but those probably need to be setup differently than what I'm doing. 
To test this I tried to change the commons-vfs-grid code to create a new GridFTPClient object for every filesystem access. But I'm not sure whether I really succeeded doing that or whether it just didn't help...

Any other suggestions?

Cheers,
Markus



----- "David Meredith (STFC,DL,ESC)" <dav...@st...> wrote:

> Hi Markus, 
> 
> Take a look at getClient() and createAndConfigureGridFtpClient() in
> GridFtpFileSystem.java. Note line 159: idleClient.changeDir(defdir);
> I'll have a crack at this later today also. 
> 
> I didn't really get very far with the many small files issue (again
> due to time issues). We also have to update to jglobus 1.7 which may
> help. I suspect that using the gridftp get and put (rather than byte
> stream) to get and put between file:// and gsiftp:// will also address
> this. 
> 
> dave 
> 
> 
> -----Original Message-----
> From: Markus Binsteiner [mailto:ma...@vp...] 
> Sent: 18 August 2009 09:26
> To: Meredith, David (STFC,DL,ESC)
> Cc: com...@li...
> Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when using
> multiple threads
> 
> Hi Dave.
> 
> Thanks for looking into that. Inline...
> 
> ----- "David Meredith (STFC,DL,ESC)" <dav...@st...>
> wrote:
> 
> > Hi Markus, 
> > 
> > I was thinking about the issues you reported last-night and I may
> > have
> > an idea why its failing. I think it may? be related to a feature
> that
> > I
> > added to the GridFTPFileSystem that prevented gridftp servers from
> > un-mounting the home directory given prolonged idle time delays
> (i.e.
> > the tcp connection stayed alive but after say 60secs idle time,
> some
> > of
> > the NGS gridftp servers would un-mount the users home dir which
> > caused
> > any subsequent VFS operations to fail). Adding this feature to punt
> > the
> > remote server to mount the users home dir if un-mounted sounds like
> a
> > potential candidate for the multi-threading issues. 
> 
> Would it be easy enough for me to try to disable this feature so I can
> test whether I get more reliabe multi-threaded transfers? Where would
> I have to look?
> 
> > The performance issue is (I think) due to the following: 
> > Presumably you are using the vfs copy function. If true, then this
> > function only uses optimisations when copying between two gridftp
> > servers for doing the 3rd party transfer. Otherwise, the method
> falls
> > back to the VFS byte streaming approach, e.g. when copying between
> > two
> > different file systems, which includes local file:// and gsiftp://.
> > This default byte streaming approach does not use any other
> protocol
> > specific optimisations to improve performance, rather it simply
> > streams
> > bytes using a input and output streams (bit pipe). Therefore, some
> > work
> > is still needed to improve this method, specifically for adding
> > switches
> > to use any optimisations for copying between local file:// and a
> > remote
> > protocol (e.g. gsiftp get and put). 
> 
> At this stage, I'm not overly concerned about the performance issues.
> At the moment I'm just piping together input & output streams myself
> (input stream from a datahandler that gets it's data from a webservice
> method) and that's probably a much bigger bottleneck than anything
> else.
> Also, I'm not really transferring that many or really big files, so I
> can live with that.
> 
> > 
> > These issues definitely need looking at, but unfortunately I am on
> > leave
> > and out of  office a great deal over the next couple of weeks
> > (however,
> > I'll take a look when I can and keep you updated regarding
> progress).
> > If
> > a proposal that I recently submitted gets funded, this will give me
> > time
> > to do all this, fingers crossed, otherwise its best effort). 
> 
> That would be great. Good luck. 
> ...to all of us :-)
> 
> BTW, anything new regarding the transfer of lots-of-small files
> issue?
> 
> Cheers,
> Markus
> 
> 
> > 
> >  
> > dave
> > 
> > 
> > 
> > 
> > -----Original Message-----
> > From: Markus Binsteiner [mailto:ma...@vp...] 
> > Sent: 17 August 2009 04:07
> > To: com...@li...
> > Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when
> using
> > multiple threads
> > 
> > Some more info:
> > 
> > First, I forgot to say: if I use only a single thread, nothing ever
> > fails.
> > 
> > Also, we did some more testing and captured some gridftp server
> logs.
> > 
> > For a connection that works, this is what the logs say:
> > 
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > PASV^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 227 Entering Passive Mode (132,181,39,23,156,66)^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > TYPE A^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 200 Type set to A.^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > CWD /home/users/bestgrid/grid-admin^M
> > [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 250 CWD command successful.^M
> > [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> > 
> > MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> >
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> > .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> > C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > 250 End.^M
> > [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368:
> > [CLIENT]:
> > PWD^M
> > [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368:
> [SERVER]:
> > 
> > 257 "/home/users/bestgrid/grid-admin" is current directory.^M
> > 
> > A similar transaction (I think) failing:
> > 
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > PASV^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 227 Entering Passive Mode (132,181,39,23,156,64)^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > TYPE A^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 200 Type set to A.^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > CWD /home/users/bestgrid/grid-admin^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 250 CWD command successful.^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [CLIENT]:
> > 
> > MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367:
> [SERVER]:
> > 
> > 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> >
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> > .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> > C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> > 250 End.^M
> > [31181] Mon Aug 17 14:33:43 2009 :: Closed connection from 
> > hqrouter.vpac.org:40367
> > 
> > So, it seems somehow the client can't interpret the server response
> > for 
> > the PWD or, more likely, the server doesn't even get the request
> > (since 
> > it doesn't say: [CLIENT]: PWD, but just closes the connection).
> > 
> > In the client debug log I have an exception like:
> > 
> > ERROR - isConnected error
> > org.globus.ftp.exception.ServerException: The server uses unknown 
> > communication protool. Custom message: (error code 2) [Nested
> > exception 
> > message: Custom message: Cannot parse 'PWD' reply: 200 Type set to
> > I.]. 
> > Nested exception is org.globus.ftp.exception.FTPReplyParseException:
> 
> > Custom message: Cannot parse 'PWD' reply: 200 Type set to I.
> > at org.globus.ftp.FTPClient.getCurrentDir(FTPClient.java:345)
> > at 
> >
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.isC
> > onnected(GridFtpFileSystem.java:176)
> > at 
> >
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.get
> > Client(GridFtpFileSystem.java:144)
> > 
> > This seems suspicious: Cannot parse 'PWD' reply: 200 Type set to I.
> > 
> > Any chance the client is mixing up connections somehow?
> > 
> > Any other ideas?
> > 
> > Cheers,
> > Markus
> > 
> > Markus Binsteiner wrote:
> > > Hi.
> > >
> > > I'm wondering whether anybody here has got experience with this:
> > >
> > > I wrote a grid job submission client that tries to make job
> > submission
> > 
> > > as easy as possible for the user. For example, it uploads files
> > from
> > the 
> > > users desktop to the jobdirectory on the cluster where the job
> will
> > run. 
> > > I'm using commons-vfs-grid to do all the gridftp things.
> > >
> > > Now, every now and then the upload of one of the input files
> fails.
> > The 
> > > reason is not always the same, sometimes its for example:
> > > Could not determine the type of file
> > "gsiftp://ng2.vpac.org/home/grid-admin"
> > > , sometimes it's:
> > > Could not write to 
> > >
> >
> "gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus
> > _Binsteiner/simpleTestTarget.txt82
> > >
> > > Sometimes it's something else...
> > >
> > > If I repeat the  same jobsubmission a few seconds later it works
> > fine.
> > >
> > > I tried to understand what's happening but it's really hard
> because
> > I 
> > > can't reproduce the errors, they just happen once in a while.
> > >
> > > To get more data, I wrote a gridftp test client which executes
> > several
> > 
> > > different tests, and one thing I can reproduce is that, if I use
> > more 
> > > threads that do the same action (e.g. upload several files at
> once
> > to 
> > > the same gridftp location), the more threads I use, the more
> errors
> > I 
> > > get. That may have nothing to do with my initial error, but it's
> > still
> > a 
> > > problem since I need to do multiple transfers in different threads
> 
> > > sometimes.
> > >
> > > Roughly, if I use 10 threads, I get about 20 failures out of 100
> > file 
> > > transmissions. Sometimes more, sometimes less. For 3 threads, I
> get
> > 
> > > around 7.
> > >
> > > I use a new FileSystem for every transfer. I set it up something
> > like:
> > >
> > > -----------------
> > >             fsmanager = VFSUtil.createNewFsManager(false, false,
> > true,
> > true,
> > >                     true, true, true, null);
> > >             // the fsmanager is created only once
> > > -----------------
> > >
> > >         FileSystemOptions opts = new FileSystemOptions();
> > >
> > >         if (fileUrl.startsWith("gsiftp")) {
> > >             GridFtpFileSystemConfigBuilder builder = 
> > > GridFtpFileSystemConfigBuilder
> > >                     .getInstance();
> > >             builder.setGSSCredential(opts, gsscredential);
> > >         }
> > >
> > >         FileObject fileRoot =
> > fsmanager.resolveFile(mp.getRootUrl(),
> > opts);
> > >
> > >         FileSystem fileBase = fileRoot.getFileSystem();
> > >
> > >        // do the transfer....
> > >
> > > Anything wrong here?
> > >
> > > This not only happens if I do simultaneous upload. It also
> happens
> > with 
> > > download or if I just do an ls on a remote directory
> > simultaneously...
> > >
> > > One theory I have is that maybe somehow the client times out more
> > easily 
> > > when there are several connections at the same time, but I
> wouldn't
> > have 
> > > a clue as to where to look for something like this.
> > >
> > > I'm not sure whether commons-vfs is threadsafe, but since I'm
> > creating
> > a 
> > > new Filesystem for every transfer that shouldn't really be an
> > issue,
> > or 
> > > am I wrong? Or is there some automatic pooling or such happening
> > down 
> > > the track either in the commons-vfs or cog libraries?
> > >
> > > Any ideas on what could be the issue? I don't think it's a server
> > error 
> > > because I used the commandline c client to do something
> equivalent
> > and
> > 
> > > that was fine (and much much faster :-) )...
> > >
> > > Cheers,
> > > Markus
> > >
> > >
> > >
> > >
> >
> ------------------------------------------------------------------------
> > ------
> > > Let Crystal Reports handle the reporting - Free Crystal Reports
> > 2008
> > 30-Day 
> > > trial. Simplify your report design, integration and deployment -
> > and
> > focus on 
> > > what you do best, core application coding. Discover what's new
> with
> > 
> > > Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> > > _______________________________________________
> > > Commonsvfsgrid-developers mailing list
> > > Com...@li...
> > >
> >
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> > >   
> > 
> > 
> >
> ------------------------------------------------------------------------
> > ------
> > Let Crystal Reports handle the reporting - Free Crystal Reports
> 2008
> > 30-Day 
> > trial. Simplify your report design, integration and deployment -
> and
> > focus on 
> > what you do best, core application coding. Discover what's new with
> 
> > Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> > _______________________________________________
> > Commonsvfsgrid-developers mailing list
> > Com...@li...
> >
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> > --
> > Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Markus B. <ma...@vp...>]

Hi Dave.

Thanks for looking into that. Inline...

----- "David Meredith (STFC,DL,ESC)" <dav...@st...> wrote:

> Hi Markus, 
> 
> I was thinking about the issues you reported last-night and I may
> have
> an idea why its failing. I think it may? be related to a feature that
> I
> added to the GridFTPFileSystem that prevented gridftp servers from
> un-mounting the home directory given prolonged idle time delays (i.e.
> the tcp connection stayed alive but after say 60secs idle time, some
> of
> the NGS gridftp servers would un-mount the users home dir which
> caused
> any subsequent VFS operations to fail). Adding this feature to punt
> the
> remote server to mount the users home dir if un-mounted sounds like a
> potential candidate for the multi-threading issues. 

Would it be easy enough for me to try to disable this feature so I can test whether I get more reliabe multi-threaded transfers? Where would I have to look?

> The performance issue is (I think) due to the following: 
> Presumably you are using the vfs copy function. If true, then this
> function only uses optimisations when copying between two gridftp
> servers for doing the 3rd party transfer. Otherwise, the method falls
> back to the VFS byte streaming approach, e.g. when copying between
> two
> different file systems, which includes local file:// and gsiftp://.
> This default byte streaming approach does not use any other protocol
> specific optimisations to improve performance, rather it simply
> streams
> bytes using a input and output streams (bit pipe). Therefore, some
> work
> is still needed to improve this method, specifically for adding
> switches
> to use any optimisations for copying between local file:// and a
> remote
> protocol (e.g. gsiftp get and put). 

At this stage, I'm not overly concerned about the performance issues. At the moment I'm just piping together input & output streams myself (input stream from a datahandler that gets it's data from a webservice method) and that's probably a much bigger bottleneck than anything else.
Also, I'm not really transferring that many or really big files, so I can live with that.

> 
> These issues definitely need looking at, but unfortunately I am on
> leave
> and out of  office a great deal over the next couple of weeks
> (however,
> I'll take a look when I can and keep you updated regarding progress).
> If
> a proposal that I recently submitted gets funded, this will give me
> time
> to do all this, fingers crossed, otherwise its best effort). 

That would be great. Good luck. 
...to all of us :-)

BTW, anything new regarding the transfer of lots-of-small files issue?

Cheers,
Markus


> 
>  
> dave
> 
> 
> 
> 
> -----Original Message-----
> From: Markus Binsteiner [mailto:ma...@vp...] 
> Sent: 17 August 2009 04:07
> To: com...@li...
> Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when using
> multiple threads
> 
> Some more info:
> 
> First, I forgot to say: if I use only a single thread, nothing ever
> fails.
> 
> Also, we did some more testing and captured some gridftp server logs.
> 
> For a connection that works, this is what the logs say:
> 
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]:
> 
> PASV^M
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]:
> 
> 227 Entering Passive Mode (132,181,39,23,156,66)^M
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]:
> 
> TYPE A^M
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]:
> 
> 200 Type set to A.^M
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]:
> 
> CWD /home/users/bestgrid/grid-admin^M
> [31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]:
> 
> 250 CWD command successful.^M
> [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [CLIENT]:
> 
> MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> [31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [SERVER]:
> 
> 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> 250 End.^M
> [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368:
> [CLIENT]:
> PWD^M
> [31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368: [SERVER]:
> 
> 257 "/home/users/bestgrid/grid-admin" is current directory.^M
> 
> A similar transaction (I think) failing:
> 
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]:
> 
> PASV^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]:
> 
> 227 Entering Passive Mode (132,181,39,23,156,64)^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]:
> 
> TYPE A^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]:
> 
> 200 Type set to A.^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]:
> 
> CWD /home/users/bestgrid/grid-admin^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]:
> 
> 250 CWD command successful.^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]:
> 
> MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> [31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]:
> 
> 250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
> .owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
> C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
> 250 End.^M
> [31181] Mon Aug 17 14:33:43 2009 :: Closed connection from 
> hqrouter.vpac.org:40367
> 
> So, it seems somehow the client can't interpret the server response
> for 
> the PWD or, more likely, the server doesn't even get the request
> (since 
> it doesn't say: [CLIENT]: PWD, but just closes the connection).
> 
> In the client debug log I have an exception like:
> 
> ERROR - isConnected error
> org.globus.ftp.exception.ServerException: The server uses unknown 
> communication protool. Custom message: (error code 2) [Nested
> exception 
> message: Custom message: Cannot parse 'PWD' reply: 200 Type set to
> I.]. 
> Nested exception is org.globus.ftp.exception.FTPReplyParseException: 
> Custom message: Cannot parse 'PWD' reply: 200 Type set to I.
> at org.globus.ftp.FTPClient.getCurrentDir(FTPClient.java:345)
> at 
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.isC
> onnected(GridFtpFileSystem.java:176)
> at 
> org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.get
> Client(GridFtpFileSystem.java:144)
> 
> This seems suspicious: Cannot parse 'PWD' reply: 200 Type set to I.
> 
> Any chance the client is mixing up connections somehow?
> 
> Any other ideas?
> 
> Cheers,
> Markus
> 
> Markus Binsteiner wrote:
> > Hi.
> >
> > I'm wondering whether anybody here has got experience with this:
> >
> > I wrote a grid job submission client that tries to make job
> submission
> 
> > as easy as possible for the user. For example, it uploads files
> from
> the 
> > users desktop to the jobdirectory on the cluster where the job will
> run. 
> > I'm using commons-vfs-grid to do all the gridftp things.
> >
> > Now, every now and then the upload of one of the input files fails.
> The 
> > reason is not always the same, sometimes its for example:
> > Could not determine the type of file
> "gsiftp://ng2.vpac.org/home/grid-admin"
> > , sometimes it's:
> > Could not write to 
> >
> "gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus
> _Binsteiner/simpleTestTarget.txt82
> >
> > Sometimes it's something else...
> >
> > If I repeat the  same jobsubmission a few seconds later it works
> fine.
> >
> > I tried to understand what's happening but it's really hard because
> I 
> > can't reproduce the errors, they just happen once in a while.
> >
> > To get more data, I wrote a gridftp test client which executes
> several
> 
> > different tests, and one thing I can reproduce is that, if I use
> more 
> > threads that do the same action (e.g. upload several files at once
> to 
> > the same gridftp location), the more threads I use, the more errors
> I 
> > get. That may have nothing to do with my initial error, but it's
> still
> a 
> > problem since I need to do multiple transfers in different threads 
> > sometimes.
> >
> > Roughly, if I use 10 threads, I get about 20 failures out of 100
> file 
> > transmissions. Sometimes more, sometimes less. For 3 threads, I get
> 
> > around 7.
> >
> > I use a new FileSystem for every transfer. I set it up something
> like:
> >
> > -----------------
> >             fsmanager = VFSUtil.createNewFsManager(false, false,
> true,
> true,
> >                     true, true, true, null);
> >             // the fsmanager is created only once
> > -----------------
> >
> >         FileSystemOptions opts = new FileSystemOptions();
> >
> >         if (fileUrl.startsWith("gsiftp")) {
> >             GridFtpFileSystemConfigBuilder builder = 
> > GridFtpFileSystemConfigBuilder
> >                     .getInstance();
> >             builder.setGSSCredential(opts, gsscredential);
> >         }
> >
> >         FileObject fileRoot =
> fsmanager.resolveFile(mp.getRootUrl(),
> opts);
> >
> >         FileSystem fileBase = fileRoot.getFileSystem();
> >
> >        // do the transfer....
> >
> > Anything wrong here?
> >
> > This not only happens if I do simultaneous upload. It also happens
> with 
> > download or if I just do an ls on a remote directory
> simultaneously...
> >
> > One theory I have is that maybe somehow the client times out more
> easily 
> > when there are several connections at the same time, but I wouldn't
> have 
> > a clue as to where to look for something like this.
> >
> > I'm not sure whether commons-vfs is threadsafe, but since I'm
> creating
> a 
> > new Filesystem for every transfer that shouldn't really be an
> issue,
> or 
> > am I wrong? Or is there some automatic pooling or such happening
> down 
> > the track either in the commons-vfs or cog libraries?
> >
> > Any ideas on what could be the issue? I don't think it's a server
> error 
> > because I used the commandline c client to do something equivalent
> and
> 
> > that was fine (and much much faster :-) )...
> >
> > Cheers,
> > Markus
> >
> >
> >
> >
> ------------------------------------------------------------------------
> ------
> > Let Crystal Reports handle the reporting - Free Crystal Reports
> 2008
> 30-Day 
> > trial. Simplify your report design, integration and deployment -
> and
> focus on 
> > what you do best, core application coding. Discover what's new with
> 
> > Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> > _______________________________________________
> > Commonsvfsgrid-developers mailing list
> > Com...@li...
> >
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> >   
> 
> 
> ------------------------------------------------------------------------
> ------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day 
> trial. Simplify your report design, integration and deployment - and
> focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Commonsvfsgrid-developers mailing list
> Com...@li...
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> --
> Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Meredith, D. (STFC,DL,ESC) <dav...@st...>]

Hi Markus, 

I was thinking about the issues you reported last-night and I may have
an idea why its failing. I think it may? be related to a feature that I
added to the GridFTPFileSystem that prevented gridftp servers from
un-mounting the home directory given prolonged idle time delays (i.e.
the tcp connection stayed alive but after say 60secs idle time, some of
the NGS gridftp servers would un-mount the users home dir which caused
any subsequent VFS operations to fail). Adding this feature to punt the
remote server to mount the users home dir if un-mounted sounds like a
potential candidate for the multi-threading issues. 

The performance issue is (I think) due to the following: 
Presumably you are using the vfs copy function. If true, then this
function only uses optimisations when copying between two gridftp
servers for doing the 3rd party transfer. Otherwise, the method falls
back to the VFS byte streaming approach, e.g. when copying between two
different file systems, which includes local file:// and gsiftp://.
This default byte streaming approach does not use any other protocol
specific optimisations to improve performance, rather it simply streams
bytes using a input and output streams (bit pipe). Therefore, some work
is still needed to improve this method, specifically for adding switches
to use any optimisations for copying between local file:// and a remote
protocol (e.g. gsiftp get and put). 

These issues definitely need looking at, but unfortunately I am on leave
and out of  office a great deal over the next couple of weeks (however,
I'll take a look when I can and keep you updated regarding progress). If
a proposal that I recently submitted gets funded, this will give me time
to do all this, fingers crossed, otherwise its best effort). 

 
dave




-----Original Message-----
From: Markus Binsteiner [mailto:ma...@vp...] 
Sent: 17 August 2009 04:07
To: com...@li...
Subject: Re: [Commonsvfsgrid-developers] Gridftp problems when using
multiple threads

Some more info:

First, I forgot to say: if I use only a single thread, nothing ever
fails.

Also, we did some more testing and captured some gridftp server logs. 
For a connection that works, this is what the logs say:

[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
PASV^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
227 Entering Passive Mode (132,181,39,23,156,66)^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
TYPE A^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
200 Type set to A.^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
CWD /home/users/bestgrid/grid-admin^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
250 CWD command successful.^M
[31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
[31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
.owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
250 End.^M
[31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368: [CLIENT]:
PWD^M
[31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
257 "/home/users/bestgrid/grid-admin" is current directory.^M

A similar transaction (I think) failing:

[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
PASV^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
227 Entering Passive Mode (132,181,39,23,156,64)^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
TYPE A^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
200 Type set to A.^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
CWD /home/users/bestgrid/grid-admin^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
250 CWD command successful.^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX
.owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
250 End.^M
[31181] Mon Aug 17 14:33:43 2009 :: Closed connection from 
hqrouter.vpac.org:40367

So, it seems somehow the client can't interpret the server response for 
the PWD or, more likely, the server doesn't even get the request (since 
it doesn't say: [CLIENT]: PWD, but just closes the connection).

In the client debug log I have an exception like:

ERROR - isConnected error
org.globus.ftp.exception.ServerException: The server uses unknown 
communication protool. Custom message: (error code 2) [Nested exception 
message: Custom message: Cannot parse 'PWD' reply: 200 Type set to I.]. 
Nested exception is org.globus.ftp.exception.FTPReplyParseException: 
Custom message: Cannot parse 'PWD' reply: 200 Type set to I.
at org.globus.ftp.FTPClient.getCurrentDir(FTPClient.java:345)
at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.isC
onnected(GridFtpFileSystem.java:176)
at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.get
Client(GridFtpFileSystem.java:144)

This seems suspicious: Cannot parse 'PWD' reply: 200 Type set to I.

Any chance the client is mixing up connections somehow?

Any other ideas?

Cheers,
Markus

Markus Binsteiner wrote:
> Hi.
>
> I'm wondering whether anybody here has got experience with this:
>
> I wrote a grid job submission client that tries to make job submission

> as easy as possible for the user. For example, it uploads files from
the 
> users desktop to the jobdirectory on the cluster where the job will
run. 
> I'm using commons-vfs-grid to do all the gridftp things.
>
> Now, every now and then the upload of one of the input files fails.
The 
> reason is not always the same, sometimes its for example:
> Could not determine the type of file
"gsiftp://ng2.vpac.org/home/grid-admin"
> , sometimes it's:
> Could not write to 
>
"gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus
_Binsteiner/simpleTestTarget.txt82
>
> Sometimes it's something else...
>
> If I repeat the  same jobsubmission a few seconds later it works fine.
>
> I tried to understand what's happening but it's really hard because I 
> can't reproduce the errors, they just happen once in a while.
>
> To get more data, I wrote a gridftp test client which executes several

> different tests, and one thing I can reproduce is that, if I use more 
> threads that do the same action (e.g. upload several files at once to 
> the same gridftp location), the more threads I use, the more errors I 
> get. That may have nothing to do with my initial error, but it's still
a 
> problem since I need to do multiple transfers in different threads 
> sometimes.
>
> Roughly, if I use 10 threads, I get about 20 failures out of 100 file 
> transmissions. Sometimes more, sometimes less. For 3 threads, I get 
> around 7.
>
> I use a new FileSystem for every transfer. I set it up something like:
>
> -----------------
>             fsmanager = VFSUtil.createNewFsManager(false, false, true,
true,
>                     true, true, true, null);
>             // the fsmanager is created only once
> -----------------
>
>         FileSystemOptions opts = new FileSystemOptions();
>
>         if (fileUrl.startsWith("gsiftp")) {
>             GridFtpFileSystemConfigBuilder builder = 
> GridFtpFileSystemConfigBuilder
>                     .getInstance();
>             builder.setGSSCredential(opts, gsscredential);
>         }
>
>         FileObject fileRoot = fsmanager.resolveFile(mp.getRootUrl(),
opts);
>
>         FileSystem fileBase = fileRoot.getFileSystem();
>
>        // do the transfer....
>
> Anything wrong here?
>
> This not only happens if I do simultaneous upload. It also happens
with 
> download or if I just do an ls on a remote directory simultaneously...
>
> One theory I have is that maybe somehow the client times out more
easily 
> when there are several connections at the same time, but I wouldn't
have 
> a clue as to where to look for something like this.
>
> I'm not sure whether commons-vfs is threadsafe, but since I'm creating
a 
> new Filesystem for every transfer that shouldn't really be an issue,
or 
> am I wrong? Or is there some automatic pooling or such happening down 
> the track either in the commons-vfs or cog libraries?
>
> Any ideas on what could be the issue? I don't think it's a server
error 
> because I used the commandline c client to do something equivalent and

> that was fine (and much much faster :-) )...
>
> Cheers,
> Markus
>
>
>
>
------------------------------------------------------------------------
------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
30-Day 
> trial. Simplify your report design, integration and deployment - and
focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Commonsvfsgrid-developers mailing list
> Com...@li...
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
>   


------------------------------------------------------------------------
------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008
30-Day 
trial. Simplify your report design, integration and deployment - and
focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Markus B. <ma...@vp...>]

Some more info:

First, I forgot to say: if I use only a single thread, nothing ever fails.

Also, we did some more testing and captured some gridftp server logs. 
For a connection that works, this is what the logs say:

[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
PASV^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
227 Entering Passive Mode (132,181,39,23,156,66)^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
TYPE A^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
200 Type set to A.^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
CWD /home/users/bestgrid/grid-admin^M
[31204] Mon Aug 17 14:33:14 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
250 CWD command successful.^M
[31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [CLIENT]: 
MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
[31204] Mon Aug 17 14:33:15 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX.owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
250 End.^M
[31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368: [CLIENT]: PWD^M
[31204] Mon Aug 17 14:33:19 2009 :: hqrouter.vpac.org:40368: [SERVER]: 
257 "/home/users/bestgrid/grid-admin" is current directory.^M

A similar transaction (I think) failing:

[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
PASV^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
227 Entering Passive Mode (132,181,39,23,156,64)^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
TYPE A^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
200 Type set to A.^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
CWD /home/users/bestgrid/grid-admin^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
250 CWD command successful.^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [CLIENT]: 
MLST C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
[31181] Mon Aug 17 14:33:13 2009 :: hqrouter.vpac.org:40367: [SERVER]: 
250-status of C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
Type=dir;Modify=20090817020959;Size=4096;Perm=cfmpel;UNIX.mode=0755;UNIX.owner=grid-admin;UNIX.group=grid-admin;Unique=f-2dcc151; 
C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner^M
250 End.^M
[31181] Mon Aug 17 14:33:43 2009 :: Closed connection from 
hqrouter.vpac.org:40367

So, it seems somehow the client can't interpret the server response for 
the PWD or, more likely, the server doesn't even get the request (since 
it doesn't say: [CLIENT]: PWD, but just closes the connection).

In the client debug log I have an exception like:

ERROR - isConnected error
org.globus.ftp.exception.ServerException: The server uses unknown 
communication protool. Custom message: (error code 2) [Nested exception 
message: Custom message: Cannot parse 'PWD' reply: 200 Type set to I.]. 
Nested exception is org.globus.ftp.exception.FTPReplyParseException: 
Custom message: Cannot parse 'PWD' reply: 200 Type set to I.
at org.globus.ftp.FTPClient.getCurrentDir(FTPClient.java:345)
at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.isConnected(GridFtpFileSystem.java:176)
at 
org.apache.commons.vfs.provider.gridftp.cogjglobus.GridFtpFileSystem.getClient(GridFtpFileSystem.java:144)

This seems suspicious: Cannot parse 'PWD' reply: 200 Type set to I.

Any chance the client is mixing up connections somehow?

Any other ideas?

Cheers,
Markus

Markus Binsteiner wrote:
> Hi.
>
> I'm wondering whether anybody here has got experience with this:
>
> I wrote a grid job submission client that tries to make job submission 
> as easy as possible for the user. For example, it uploads files from the 
> users desktop to the jobdirectory on the cluster where the job will run. 
> I'm using commons-vfs-grid to do all the gridftp things.
>
> Now, every now and then the upload of one of the input files fails. The 
> reason is not always the same, sometimes its for example:
> Could not determine the type of file "gsiftp://ng2.vpac.org/home/grid-admin"
> , sometimes it's:
> Could not write to 
> "gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner/simpleTestTarget.txt82
>
> Sometimes it's something else...
>
> If I repeat the  same jobsubmission a few seconds later it works fine.
>
> I tried to understand what's happening but it's really hard because I 
> can't reproduce the errors, they just happen once in a while.
>
> To get more data, I wrote a gridftp test client which executes several 
> different tests, and one thing I can reproduce is that, if I use more 
> threads that do the same action (e.g. upload several files at once to 
> the same gridftp location), the more threads I use, the more errors I 
> get. That may have nothing to do with my initial error, but it's still a 
> problem since I need to do multiple transfers in different threads 
> sometimes.
>
> Roughly, if I use 10 threads, I get about 20 failures out of 100 file 
> transmissions. Sometimes more, sometimes less. For 3 threads, I get 
> around 7.
>
> I use a new FileSystem for every transfer. I set it up something like:
>
> -----------------
>             fsmanager = VFSUtil.createNewFsManager(false, false, true, true,
>                     true, true, true, null);
>             // the fsmanager is created only once
> -----------------
>
>         FileSystemOptions opts = new FileSystemOptions();
>
>         if (fileUrl.startsWith("gsiftp")) {
>             GridFtpFileSystemConfigBuilder builder = 
> GridFtpFileSystemConfigBuilder
>                     .getInstance();
>             builder.setGSSCredential(opts, gsscredential);
>         }
>
>         FileObject fileRoot = fsmanager.resolveFile(mp.getRootUrl(), opts);
>
>         FileSystem fileBase = fileRoot.getFileSystem();
>
>        // do the transfer....
>
> Anything wrong here?
>
> This not only happens if I do simultaneous upload. It also happens with 
> download or if I just do an ls on a remote directory simultaneously...
>
> One theory I have is that maybe somehow the client times out more easily 
> when there are several connections at the same time, but I wouldn't have 
> a clue as to where to look for something like this.
>
> I'm not sure whether commons-vfs is threadsafe, but since I'm creating a 
> new Filesystem for every transfer that shouldn't really be an issue, or 
> am I wrong? Or is there some automatic pooling or such happening down 
> the track either in the commons-vfs or cog libraries?
>
> Any ideas on what could be the issue? I don't think it's a server error 
> because I used the commandline c client to do something equivalent and 
> that was fine (and much much faster :-) )...
>
> Cheers,
> Markus
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Commonsvfsgrid-developers mailing list
> Com...@li...
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
>   

[Commonsvfsgrid-developers] Gridftp problems when using multiple threads

[Markus B. <ma...@vp...>]

Hi.

I'm wondering whether anybody here has got experience with this:

I wrote a grid job submission client that tries to make job submission 
as easy as possible for the user. For example, it uploads files from the 
users desktop to the jobdirectory on the cluster where the job will run. 
I'm using commons-vfs-grid to do all the gridftp things.

Now, every now and then the upload of one of the input files fails. The 
reason is not always the same, sometimes its for example:
Could not determine the type of file "gsiftp://ng2.vpac.org/home/grid-admin"
, sometimes it's:
Could not write to 
"gsiftp://ng2.vpac.org/home/grid-admin/C_AU_O_APACGrid_OU_VPAC_CN_Markus_Binsteiner/simpleTestTarget.txt82

Sometimes it's something else...

If I repeat the  same jobsubmission a few seconds later it works fine.

I tried to understand what's happening but it's really hard because I 
can't reproduce the errors, they just happen once in a while.

To get more data, I wrote a gridftp test client which executes several 
different tests, and one thing I can reproduce is that, if I use more 
threads that do the same action (e.g. upload several files at once to 
the same gridftp location), the more threads I use, the more errors I 
get. That may have nothing to do with my initial error, but it's still a 
problem since I need to do multiple transfers in different threads 
sometimes.

Roughly, if I use 10 threads, I get about 20 failures out of 100 file 
transmissions. Sometimes more, sometimes less. For 3 threads, I get 
around 7.

I use a new FileSystem for every transfer. I set it up something like:

-----------------
            fsmanager = VFSUtil.createNewFsManager(false, false, true, true,
                    true, true, true, null);
            // the fsmanager is created only once
-----------------

        FileSystemOptions opts = new FileSystemOptions();

        if (fileUrl.startsWith("gsiftp")) {
            GridFtpFileSystemConfigBuilder builder = 
GridFtpFileSystemConfigBuilder
                    .getInstance();
            builder.setGSSCredential(opts, gsscredential);
        }

        FileObject fileRoot = fsmanager.resolveFile(mp.getRootUrl(), opts);

        FileSystem fileBase = fileRoot.getFileSystem();

       // do the transfer....

Anything wrong here?

This not only happens if I do simultaneous upload. It also happens with 
download or if I just do an ls on a remote directory simultaneously...

One theory I have is that maybe somehow the client times out more easily 
when there are several connections at the same time, but I wouldn't have 
a clue as to where to look for something like this.

I'm not sure whether commons-vfs is threadsafe, but since I'm creating a 
new Filesystem for every transfer that shouldn't really be an issue, or 
am I wrong? Or is there some automatic pooling or such happening down 
the track either in the commons-vfs or cog libraries?

Any ideas on what could be the issue? I don't think it's a server error 
because I used the commandline c client to do something equivalent and 
that was fine (and much much faster :-) )...

Cheers,
Markus

Re: [Commonsvfsgrid-developers] Setting timeout for gridftp filesystem

[Meredith, DJ (David) <dav...@st...>]

Hi all, 
Ok, I added griftp timeout support. Also, I deleted
'src/test/resources/log4j.properties' file as this file is ignored (the
'src/main/resources/log4j.properties' is not). 
I'll take a look at the gridftp small files thingy. 

Dave





-----Original Message-----
From: Markus Binsteiner [mailto:ma...@vp...] 
Sent: 01 July 2009 10:40
To: com...@li...
Subject: Re: [Commonsvfsgrid-developers] Setting timeout for gridftp
filesystem

----- "DJ Meredith (David)" <dav...@st...> wrote:
> 
> I'll add gridftp timeout support today. 
> Ps, Markus, I would also like to look at the small files issue we
> discussed recently. Could you point me in the right direction to the
> web
> pages where you saw that the jglobus/cog had addressed this issue ? 

Hi Dave.

That's where I say the globus guys mentioning the small-files thing:
http://dev.globus.org/wiki/CoG_JGlobus_1.5.0

"GridFTP API support for pipelining" 
Don't know how it works and where they documented how to do it, though. 

Thanks heaps for looking into these issues. I really appreciate it.

BTW, you wanted to talk about job templates at one stage. If you're
still interested, just tell me a when...

Cheers,
Markus

> 
> Dave
> 
> 
> 
> -----Original Message-----
> From: Markus Binsteiner [mailto:ma...@vp...] 
> Sent: 29 June 2009 04:00
> To: com...@li...
> Subject: [Commonsvfsgrid-developers] Setting timeout for gridftp
> filesystem
> 
> Hi guys.
> 
> I was wondering: is there a way to specify a timeout for a 
> gridftp-connection when using the commons-vfs-grid plugin? I am having
> 
> problems accessing some of the filesystems mds says that are
> available, 
> but sometimes a firewall just drops the packets and my client just
> sits 
> and waits.
> 
> At the moment it takes about a minute or two until I get an error 
> message. This is a bit too long for a user interface. So, is there a
> way
> 
> to reduce this time?
> 
> Thanks in advance.
> 
> Cheers,
> Markus
> 
>
------------------------------------------------------------------------
> ------
> _______________________________________________
> Commonsvfsgrid-developers mailing list
> Com...@li...
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> --
> Scanned by iCritical.

------------------------------------------------------------------------
------
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Setting timeout for gridftp filesystem

[Markus B. <ma...@vp...>]

----- "DJ Meredith (David)" <dav...@st...> wrote:
> 
> I'll add gridftp timeout support today. 
> Ps, Markus, I would also like to look at the small files issue we
> discussed recently. Could you point me in the right direction to the
> web
> pages where you saw that the jglobus/cog had addressed this issue ? 

Hi Dave.

That's where I say the globus guys mentioning the small-files thing:
http://dev.globus.org/wiki/CoG_JGlobus_1.5.0

"GridFTP API support for pipelining" 
Don't know how it works and where they documented how to do it, though. 

Thanks heaps for looking into these issues. I really appreciate it.

BTW, you wanted to talk about job templates at one stage. If you're still interested, just tell me a when...

Cheers,
Markus

> 
> Dave
> 
> 
> 
> -----Original Message-----
> From: Markus Binsteiner [mailto:ma...@vp...] 
> Sent: 29 June 2009 04:00
> To: com...@li...
> Subject: [Commonsvfsgrid-developers] Setting timeout for gridftp
> filesystem
> 
> Hi guys.
> 
> I was wondering: is there a way to specify a timeout for a 
> gridftp-connection when using the commons-vfs-grid plugin? I am having
> 
> problems accessing some of the filesystems mds says that are
> available, 
> but sometimes a firewall just drops the packets and my client just
> sits 
> and waits.
> 
> At the moment it takes about a minute or two until I get an error 
> message. This is a bit too long for a user interface. So, is there a
> way
> 
> to reduce this time?
> 
> Thanks in advance.
> 
> Cheers,
> Markus
> 
> ------------------------------------------------------------------------
> ------
> _______________________________________________
> Commonsvfsgrid-developers mailing list
> Com...@li...
> https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
> --
> Scanned by iCritical.

Re: [Commonsvfsgrid-developers] Setting timeout for gridftp filesystem

[Meredith, DJ (David) <dav...@st...>]

Hi folks, 
Apologies for the late response - I have been out of office for a while.

I'll add gridftp timeout support today. 
Ps, Markus, I would also like to look at the small files issue we
discussed recently. Could you point me in the right direction to the web
pages where you saw that the jglobus/cog had addressed this issue ? 

Dave



-----Original Message-----
From: Markus Binsteiner [mailto:ma...@vp...] 
Sent: 29 June 2009 04:00
To: com...@li...
Subject: [Commonsvfsgrid-developers] Setting timeout for gridftp
filesystem

Hi guys.

I was wondering: is there a way to specify a timeout for a 
gridftp-connection when using the commons-vfs-grid plugin? I am having 
problems accessing some of the filesystems mds says that are available, 
but sometimes a firewall just drops the packets and my client just sits 
and waits.

At the moment it takes about a minute or two until I get an error 
message. This is a bit too long for a user interface. So, is there a way

to reduce this time?

Thanks in advance.

Cheers,
Markus

------------------------------------------------------------------------
------
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

[Commonsvfsgrid-developers] Setting timeout for gridftp filesystem

[Markus B. <ma...@vp...>]

Hi guys.

I was wondering: is there a way to specify a timeout for a 
gridftp-connection when using the commons-vfs-grid plugin? I am having 
problems accessing some of the filesystems mds says that are available, 
but sometimes a firewall just drops the packets and my client just sits 
and waits.

At the moment it takes about a minute or two until I get an error 
message. This is a bit too long for a user interface. So, is there a way 
to reduce this time?

Thanks in advance.

Cheers,
Markus

Re: [Commonsvfsgrid-developers] More updates

[Pauline M. <Pau...@ar...>]

Hi Dave,

Meredith, DJ (David) wrote:
> Hi Pauline, 
> 
> I will take a look at the gridftp jglobus1.6 stuff next week, sorry this
> can't be sooner but I am rushing trying to finish off some other stuff
> at the mo. A couple of follow on questions: 
> 
> 1) Did you manage to resolve the 3rd party file transfer issues for SRB
> (i.e. byte streaming from srb/irods to/from another provider)? I
> remember that this was causing you some problems. If true (i.e. fixed),
> in short what was the issue ? 

Woops,  I don't think I have answered your question - I think the SRB 
provider is fixed... but as to what the exact problem was, I can't 
really tell.  I'd spent ages changing the code around... I had something 
to do with the caching - things were just not being updated.  I think 
something in onChildrenChange wasn't calling the right methods.

I'm now having issues with iRODS folders - iRODS seems to think I have 
created files instead... but that's another can of worms...


Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/

Re: [Commonsvfsgrid-developers] More updates

[Pauline M. <Pau...@ar...>]

Hi Dave,

Ok, good to hear that GridFTP is working for you.  For some reason, I 
can't do GridFTP 3rd party transfer between two hosts.  I've tested the 
connection using globus-url-copy and it works, but when I used 
commons-vfs-grid, I get a connection time out in *one* direction.  And I 
have no idea on how to track down what the problem is...

Anyway... the small files problem:

http://dev.globus.org/wiki/CoG_JGlobus_1.5.0

There's a sentence on:

"GridFTP API support for pipelining. A performance improvement for the 
"Lots of Small Files" problem. "

At the moment, Markus is building a something called Grisu that will do 
some file transfers on gridded protocols.  One of the use case is 
transferring many small files.  20M would take up to 20 minutes, which 
isn't ideal...  It has been fixed in 1.5.0, but doesn't seem to work 
under the 3rd party transfer stuff.  I could be as simple as changing a 
method call, but neither of us know much about globus...  If you can 
take a look, that would be great :)  Markus should be on this mailing 
list too... so I guess he can elaborate (I'm only giving second hand info!)

Cheers,

-Pauline.

Meredith, DJ (David) wrote:
> Hi Pauline, 
> 
> I think I wrongly interpreted your last email to mean there was
> something is wrong with the gridftp provider, apologies (its all still
> working good for me). I would be happy to help Markus to add any new
> features, can Markus expand on the extra support for small file
> transfers ? 
> 
> Cheers
> Dave
> 
> 
> -----Original Message-----
> From: Pauline Mak [mailto:Pau...@ar...] 
> Sent: 04 June 2009 07:43
> To: com...@li...
> Subject: [Commonsvfsgrid-developers] More updates
> 
> Hi all,
> 
> Just a few things:
> 
> I've updated the source again with a few modiifications.  I've added in 
> Jargon 2.0.5 that's been recompiled with java 1.5 (mainly because Macs 
> don't officially support 1.6 yet *sigh*).  So update your source and 
> recompile should download the right dependencies :)
> 
> I'm slowly copying a few of the test classes across to iRODS and is 
> working surprisingly well!
> 
> Finally - Markus is hoping to add a feature into the GridFTP provider - 
> the ability to transfer lots of small files quickly.  Apparently this 
> was fixed in jglobus-1.5.0, but that's only working for direct transfer 
> and not for 3rd party transfer.  Dave - would you be able to help with 
> this?
> 
> Cheers,
> 
> -Pauline.
> 


-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/

Re: [Commonsvfsgrid-developers] More updates

[Meredith, DJ (David) <dav...@st...>]

Hi Pauline, 

I think I wrongly interpreted your last email to mean there was
something is wrong with the gridftp provider, apologies (its all still
working good for me). I would be happy to help Markus to add any new
features, can Markus expand on the extra support for small file
transfers ? 

Cheers
Dave


-----Original Message-----
From: Pauline Mak [mailto:Pau...@ar...] 
Sent: 04 June 2009 07:43
To: com...@li...
Subject: [Commonsvfsgrid-developers] More updates

Hi all,

Just a few things:

I've updated the source again with a few modiifications.  I've added in 
Jargon 2.0.5 that's been recompiled with java 1.5 (mainly because Macs 
don't officially support 1.6 yet *sigh*).  So update your source and 
recompile should download the right dependencies :)

I'm slowly copying a few of the test classes across to iRODS and is 
working surprisingly well!

Finally - Markus is hoping to add a feature into the GridFTP provider - 
the ability to transfer lots of small files quickly.  Apparently this 
was fixed in jglobus-1.5.0, but that's only working for direct transfer 
and not for 3rd party transfer.  Dave - would you be able to help with 
this?

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/




------------------------------------------------------------------------
------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
looking to deploy the next generation of Solaris that includes the
latest 
innovations from Sun and the OpenSource community. Download a copy and 
enjoy capabilities such as Networking, Storage and Virtualization. 
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

Re: [Commonsvfsgrid-developers] More updates

[Meredith, DJ (David) <dav...@st...>]

Hi Pauline, 

I will take a look at the gridftp jglobus1.6 stuff next week, sorry this
can't be sooner but I am rushing trying to finish off some other stuff
at the mo. A couple of follow on questions: 

1) Did you manage to resolve the 3rd party file transfer issues for SRB
(i.e. byte streaming from srb/irods to/from another provider)? I
remember that this was causing you some problems. If true (i.e. fixed),
in short what was the issue ? 
2) Can you clarify what you mean when you say that the small file
transfer resolution only works for direct transfer and not for 3rd party
transfer? (me being slow, but when you mean direct transfer do you mean
the file streaming approach used for copying between different providers
or the true 3rd party transfer that applies for two gridftp providers
only ?). 

Cheers, 
Dave
 



-----Original Message-----
From: Pauline Mak [mailto:Pau...@ar...] 
Sent: 04 June 2009 07:43
To: com...@li...
Subject: [Commonsvfsgrid-developers] More updates

Hi all,

Just a few things:

I've updated the source again with a few modiifications.  I've added in 
Jargon 2.0.5 that's been recompiled with java 1.5 (mainly because Macs 
don't officially support 1.6 yet *sigh*).  So update your source and 
recompile should download the right dependencies :)

I'm slowly copying a few of the test classes across to iRODS and is 
working surprisingly well!

Finally - Markus is hoping to add a feature into the GridFTP provider - 
the ability to transfer lots of small files quickly.  Apparently this 
was fixed in jglobus-1.5.0, but that's only working for direct transfer 
and not for 3rd party transfer.  Dave - would you be able to help with 
this?

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/




------------------------------------------------------------------------
------
OpenSolaris 2009.06 is a cutting edge operating system for enterprises 
looking to deploy the next generation of Solaris that includes the
latest 
innovations from Sun and the OpenSource community. Download a copy and 
enjoy capabilities such as Networking, Storage and Virtualization. 
Go to: http://p.sf.net/sfu/opensolaris-get
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

[Commonsvfsgrid-developers] More updates

[Pauline M. <Pau...@ar...>]

Hi all,

Just a few things:

I've updated the source again with a few modiifications.  I've added in 
Jargon 2.0.5 that's been recompiled with java 1.5 (mainly because Macs 
don't officially support 1.6 yet *sigh*).  So update your source and 
recompile should download the right dependencies :)

I'm slowly copying a few of the test classes across to iRODS and is 
working surprisingly well!

Finally - Markus is hoping to add a feature into the GridFTP provider - 
the ability to transfer lots of small files quickly.  Apparently this 
was fixed in jglobus-1.5.0, but that's only working for direct transfer 
and not for 3rd party transfer.  Dave - would you be able to help with 
this?

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/

Re: [Commonsvfsgrid-developers] Trunk updated

[Meredith, DJ (David) <dav...@st...>]

Hi Pauline, 

I'll take a look at the gridftp issues. 
Dave


-----Original Message-----
From: Pauline Mak [mailto:Pau...@ar...] 
Sent: 31 May 2009 15:32
To: com...@li...
Subject: [Commonsvfsgrid-developers] Trunk updated

Hi all,

Just letting you know that I have committed some code to the trunk!  I 
think the crux of the code reshuffle is done now.  It passes most of the

tests as far as I can tell, but I'm having some problems with the 
GridFTP 3rd party transfers.  I really hope this has nothing to do with 
the updated globus libraries (which are now all 1.6.0).

I think I have passed the test file transfer between SRB and other FS. 
Let me know if that is not the case!  I've made some pretty silly errors

along the way, which cost me dearly time-wise :(

Feel free to give it a test too!  More updates when I write some of 
those iRODS tests....

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/




------------------------------------------------------------------------
------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT 
is a gathering of tech-side developers & brand creativity professionals.
Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp as they present alongside digital heavyweights like
Barbarian 
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers
-- 
Scanned by iCritical.

[Commonsvfsgrid-developers] Trunk updated

[Pauline M. <Pau...@ar...>]

Hi all,

Just letting you know that I have committed some code to the trunk!  I 
think the crux of the code reshuffle is done now.  It passes most of the 
tests as far as I can tell, but I'm having some problems with the 
GridFTP 3rd party transfers.  I really hope this has nothing to do with 
the updated globus libraries (which are now all 1.6.0).

I think I have passed the test file transfer between SRB and other FS. 
Let me know if that is not the case!  I've made some pretty silly errors 
along the way, which cost me dearly time-wise :(

Feel free to give it a test too!  More updates when I write some of 
those iRODS tests....

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/

Re: [Commonsvfsgrid-developers] Update

[Meredith, DJ \(David\) <dav...@st...>]

Hi Pauline, 

The VFS caching sounds to me like the most likely suspect. I think to
force clearing the cache you may need to override the doDetach() method
in the FileObjectImplementation, but I am sure you have probably tried
this already (example below taken from my old srb implementation in the
djm76import branch). 

   /**
     * Detaches this FileObject from its file resource
     * Is Called when this file is closed.
     * Note that this FileObject may be reused later, so should be able
to be reattached
     * 
     * @see
org.apache.commons.vfs.provider.AbstractFileObject#doDetach()
     */
    @Override
    protected void doDetach() {
        // If not overidden, default impl does nothing.
        log.debug("doDetach() of: " + getName().getPath());
        //nullifying fileInfo and children loses the cache
        this.file = null;
        this.children = null;
    }


I'll take a look at the todo list.

Dave


-----Original Message-----
From: Pauline Mak [mailto:Pau...@ar...] 
Sent: 21 May 2009 01:00
To: com...@li...
Subject: [Commonsvfsgrid-developers] Update

Hi all,

Many apologies for not sending an update email earlier.

I've been busy rejigging the commons-vfs-grid code to bring the SRB and 
iRODS providers more inline with each other.  That broke quite a few 
things... but it is passing all the tests for providers.  I am still 
trying to track down the problem with copying files between SRB and 
other files systems - I suspect this has something to do with the 
caching mechanism in the SRB provider is not being refreshed properly 
when file changes.  Hopefully I'll find it soon (or else, I'll lose my 
sanity!)

Things are still pending:
- iRODS tests (sorry... it's still not written yet)
- resume transfers (as suggested by Markus)
- tilde in URIs


I'm still hoping for a release of 1.5 sometime in June, with at least 
the iRODS tests, but time is ticking rather quickly.

Dave - have all the other tasks been addressed?  I'm going through an 
email from 4th of March which had 4 things to do, and I believe the 
gridFTP implementation has been addressed and the 
SRBFileSystemConfigBuilder has been changed (although, not committed to 
SVN yet!)  The tests are also in the code now.

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/




------------------------------------------------------------------------
------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals.
Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp asthey present alongside digital heavyweights like
Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
Commonsvfsgrid-developers mailing list
Com...@li...
https://lists.sourceforge.net/lists/listinfo/commonsvfsgrid-developers

[Commonsvfsgrid-developers] Update

[Pauline M. <Pau...@ar...>]

Hi all,

Many apologies for not sending an update email earlier.

I've been busy rejigging the commons-vfs-grid code to bring the SRB and 
iRODS providers more inline with each other.  That broke quite a few 
things... but it is passing all the tests for providers.  I am still 
trying to track down the problem with copying files between SRB and 
other files systems - I suspect this has something to do with the 
caching mechanism in the SRB provider is not being refreshed properly 
when file changes.  Hopefully I'll find it soon (or else, I'll lose my 
sanity!)

Things are still pending:
- iRODS tests (sorry... it's still not written yet)
- resume transfers (as suggested by Markus)
- tilde in URIs


I'm still hoping for a release of 1.5 sometime in June, with at least 
the iRODS tests, but time is ticking rather quickly.

Dave - have all the other tasks been addressed?  I'm going through an 
email from 4th of March which had 4 things to do, and I believe the 
gridFTP implementation has been addressed and the 
SRBFileSystemConfigBuilder has been changed (although, not committed to 
SVN yet!)  The tests are also in the code now.

Cheers,

-Pauline.

-- 
Pauline Mak

ARCS Data Services
Ph: (03) 6226 7518
Email: pau...@ar...
Jabber: pau...@ar...
http://www.arcs.org.au/

TPAC
Email: pau...@ut...
http://www.tpac.org.au/