soaplab-users

[Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

Hi,

I have (now) a problem with an old application which doesn't seem to 
accept path provided by SoapLab2 for input file:
-in=/opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]3bd3a626.118f08a1fec.-7ffe/i_infile

Do you have any idea how to deal with this ?
I have tried this syntax to protect the path but without result:
	template: "-in=\"$$\""

Thanks

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> Do you have any idea how to deal with this ?


If the file is in the SANDBOX it should be fine. I have just looked into the
code and it seems to confirm what I have just said but obviously it does not
work that way (otherwise you would not have this error). Again, I need to
run some tests on this - which I will do the first thing in (my) morning.

We had this issues before (it was for EMBOSS jobs) and I remember that we
have decided to allow (by setting a property) to overwrite this SANDBOX
location - but I have never implemented it. I may do it now when looking
into it.

I will let you know asap.
Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> I have (now) a problem with an old application which doesn't seem to
> accept path provided by SoapLab2 for input file:
>
> -in=/opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]3bd3a626.118f08a1fec.-7ffe/i_infile
>

I have tried myself, and I cannot reproduce the error here.

Could you checks if the path mentioned in your example is a path of the
SANDBOX of your current Soaplab2? Because Soaplab2 refuses to take local
(local on the server side) files unless they are in its SANDBOX directory.
What is the full error message you are getting (something like "Refusing to
resolve the reference...")?

Could you try to run a testing service (which comes with Soaplab2) - using
as an input an existing file in your SANDBOX (the example below is just from
my machine):

build/run/run-cmdline-client -protocol local -name testing.files -w -r
input_url /home/senger/soaplab2/_R_/SANDBOX/\[classic.helloworld\]-
24c8efb0.117ea7cd13f.-8000/o_output

This calls works fine on my machine, how about on yours?

Thanks,
Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     I have (now) a problem with an old application which doesn't seem to
>     accept path provided by SoapLab2 for input file:
>     -in=/opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]3bd3a626.118f08a1fec.-7ffe/i_infile
> 
> 
> I have tried myself, and I cannot reproduce the error here.
> 
> Could you checks if the path mentioned in your example is a path of the 
> SANDBOX of your current Soaplab2? Because Soaplab2 refuses to take local 
> (local on the server side) files unless they are in its SANDBOX 
> directory. What is the full error message you are getting (something 
> like "Refusing to resolve the reference...")?

The file is exactly where mentioned by the path. It is readable and has 
the proper syntax.

It's my application which cannot read the file with this path.
It doesn't read it in command line, except if I use a path without 
special characters ( [ ] - ), or if I change path to 
/opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]3bd3a626.118f08a1fec.-7ffe/
and run -in=./i_infile


So, I am looking for a way to protect the path, or to avoid special 
characters inside it.
Something like 	template: "-in=\"$$\""
or		template: "-in='$$'"
but it failed.

> Could you try to run a testing service (which comes with Soaplab2) - 
> using as an input an existing file in your SANDBOX (the example below is 
> just from my machine):
> 
> build/run/run-cmdline-client -protocol local -name testing.files -w -r 
> input_url 
> /home/senger/soaplab2/_R_/SANDBOX/\[classic.helloworld\]-24c8efb0.117ea7cd13f.-8000/o_output

Other services, like EMBOSS services with infile, work fine.

> This calls works fine on my machine, how about on yours?
> 
> Thanks,
> Martin

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> It's my application which cannot read the file with this path.
> It doesn't read it in command line, except if I use a path without
> special characters ( [ ] - )


Okay, I understand, but your application is running on your machine, it is a
Soaplab's client. Once this client application sends a filename to a
Soplab's service, things should be fine (first, because there is no shell
involved on the Soaplab side, and second, because it work fine for other
Soaplab services, as shown in the example in my previous email).


> Something like  template: "-in=\"$$\""
> or              template: "-in='$$'"
> but it failed.
>

Again, we are now talking about a client application, so it has nothing to
do with ACD files - therefore using different templates cannot hep here.

What can help is to deal with your shell. If you use bash, it does not have
problems with [ and ], only if you use tcsh you need to escape (or quote)
these characters. What shell are you using?

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     It's my application which cannot read the file with this path.
>     It doesn't read it in command line, except if I use a path without
>     special characters ( [ ] - )
> 
> 
> Okay, I understand, but your application is running on your machine, it 
> is a Soaplab's client. Once this client application sends a filename to 
> a Soplab's service, things should be fine (first, because there is no 
> shell involved on the Soaplab side, and second, because it work fine for 
> other Soaplab services, as shown in the example in my previous email).
>  
> 
>     Something like  template: "-in=\"$$\""
>     or              template: "-in='$$'"
>     but it failed.
> 
> 
> Again, we are now talking about a client application, so it has nothing 
> to do with ACD files - therefore using different templates cannot hep here.
> 
> What can help is to deal with your shell. If you use bash, it does not 
> have problems with [ and ], only if you use tcsh you need to escape (or 
> quote) these characters. What shell are you using?
> 
> Martin

I know it is not the ACD syntax but the shell which is involved because 
I have the same problem in command line with these paths.

I use bash and it is my application which seems to be the real problem.
It cannot read the file from this path even if special characters are 
escaped.


It is an old application (mid 90's) and I wonder if ACD syntax could 
protect path for my application.

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> I use bash and it is my application which seems to be the real problem.
> It cannot read the file from this path even if special characters are
> escaped.


Just please confirm that I understand it correctly: Your application is a
client that sends data to a Soaplab service, right?

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     I use bash and it is my application which seems to be the real problem.
>     It cannot read the file from this path even if special characters are
>     escaped.
> 
> 
> Just please confirm that I understand it correctly: Your application is 
> a client that sends data to a Soaplab service, right?
> 
> Martin

No, my application is the Soaplab service.

Here is the output I get:
sequence file not found
Runtime error 2 at $08062B7E
   $08062B7E
   $080638B6
   $08066193
   $0804BCD7
   $0804C192
   $0804809F


I have just tried to rename
/opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]4a976779.118f46927cf.-7ffe/
to
/opt/tomcat/temp/_R_/SANDBOX/protein_2d_structure.tmpred_4a976779.118f46927cf.7ffe/

and now 
-in=/opt/tomcat/temp/_R_/SANDBOX/protein_2d_structure.tmpred_4a976779.118f46927cf.7ffe/i_infile
works

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> No, my application is the Soaplab service.


If you application is the one called by a Soaplab service, then it gets data
without using any shell. Therefore, it is, indeed, your application that is
not able to use a parameter that has special characters.

I have just tried to rename
>
> /opt/tomcat/temp/_R_/SANDBOX/[protein_2d_structure.tmpred]4a976779.118f46927cf.-7ffe/


No, you cannot rename it: this file name (and the file itself) is created
every time a Soaplab service is called. You do not have access to a file to
rename it in the right time.

You need to look into your application how it parses and uses parameters. If
you cannot do it (because it's too old or not well readable etc.) you need
to write a script that will warp it - and this script will read the file
name given to it by Soaplab, will copy the fiole to a temporary directory
and will pass a new name to your application itself (together with the other
parameters gotten from the Soaplab service).

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Mahmut U. <ul...@eb...>]

> you need to write a script that will warp it - and this script will
> read the file name given to it by Soaplab, will copy the file to a
> temporary directory and will pass a new name to your application
> itself (together with the other parameters gotten from the Soaplab
> service).

One other option would be to write an input data adaptor class in Java
that should implement the interface
org.soaplab.services.adaptor.DataAdaptor. However, for your case writing
a wrapper seems to be a more practical solution.

Mahmut

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> One other option would be to write an input data adaptor class in Java
> that should implement the interface
> org.soaplab.services.adaptor.DataAdaptor.


I am afraid that it may not be true. The adaptors are in charge when data
arrives from a user and before the Soaplab service processes them. But
Sebastien's problem occurs when Soaplab calls an external, command-line,
application. In that moment all input data are already "adapted" (if there
was any adaptor for them) and they sit in a Soaplab's internal file whose
name is passed to the external application. And here the name is the problem
- not the data inside the file.

Cheers,
Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Mahmut U. <ul...@eb...>]

Hi Martin,

>         One other option would be to write an input data adaptor class
>         in Java
>         that should implement the interface
>         org.soaplab.services.adaptor.DataAdaptor.
> 
> I am afraid that it may not be true. The adaptors are in charge when
> data arrives from a user and before the Soaplab service processes
> them. But Sebastien's problem occurs when Soaplab calls an external,
> command-line, application. In that moment all input data are already
> "adapted" (if there was any adaptor for them) and they sit in a
> Soaplab's internal file whose name is passed to the external
> application. And here the name is the problem - not the data inside
> the file.

I was looking at the InputManyFiles adaptor (for some EMBASSY
applications which expects directory name rather than list of file
names), given some input objects it returns a string with the list of
input file names. It seems Soaplab should not make problem if this
adaptor returns a string without these special characters.

Meantime, I noticed the following method in echo.acd.

#   comment: 'method foreachs (str, " ", "-&&# $str")'

I don't know much about the substitution support in Soaplab. Is it
possible to use the existing substitution support for this problem?

Mahmut

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

> Hi Martin,
> 
>>         One other option would be to write an input data adaptor class
>>         in Java
>>         that should implement the interface
>>         org.soaplab.services.adaptor.DataAdaptor.
>>
>> I am afraid that it may not be true. The adaptors are in charge when
>> data arrives from a user and before the Soaplab service processes
>> them. But Sebastien's problem occurs when Soaplab calls an external,
>> command-line, application. In that moment all input data are already
>> "adapted" (if there was any adaptor for them) and they sit in a
>> Soaplab's internal file whose name is passed to the external
>> application. And here the name is the problem - not the data inside
>> the file.
> 
> I was looking at the InputManyFiles adaptor (for some EMBASSY
> applications which expects directory name rather than list of file
> names), given some input objects it returns a string with the list of
> input file names. It seems Soaplab should not make problem if this
> adaptor returns a string without these special characters.
> 
> Meantime, I noticed the following method in echo.acd.
> 
> #   comment: 'method foreachs (str, " ", "-&&# $str")'
> 
> I don't know much about the substitution support in Soaplab. Is it
> possible to use the existing substitution support for this problem?
> 
> Mahmut

I made several tests and it is clearly a problem from my application.
The path works fine in interactive mode but not in command line !

So, is there a way to do a kind of 'cd path/' and then to run 
applications with infiles and outfiles from the current directory.

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> So, is there a way to do a kind of 'cd path/' and then to run
> applications with infiles and outfiles from the current directory.
>

Before it starts an external application, Soaplab2 changes its working
directory (making a 'cd') into the job directory (which is a directory
.../SANDBOX/[job-name]/

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     So, is there a way to do a kind of 'cd path/' and then to run
>     applications with infiles and outfiles from the current directory.
> 
> 
> Before it starts an external application, Soaplab2 changes its working 
> directory (making a 'cd') into the job directory (which is a directory 
> .../SANDBOX/[job-name]/
> 
> Martin

Nevertheless, service report file shows the full path
-in=/opt/tomcat/temp/_R_/SANDBOX/[embnet.tmpred]-47cb20aa.118f4f77748.-7fff/i_infile

So, external applications use full paths or relative paths ?

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> So, external applications use full paths or relative paths ?


They are getting from Soaplab always full paths.

M.


-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     So, external applications use full paths or relative paths ?
> 
> 
> They are getting from Soaplab always full paths.
> 
> M.

More precisely, this is '-', in the path, which makes my program crashed.

Where can I replace '-' by '_' in the soaplab2 sources ?

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> More precisely, this is '-', in the path, which makes my program crashed.
>
> Where can I replace '-' by '_' in the soaplab2 sources ?
>

The name of the directory in question is also the job ID. It is created in
org.soaplab.services.JobManager in method createJobId(). The minus is given
by calling 'new java.rmi.server.UID().toString()'.

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>     More precisely, this is '-', in the path, which makes my program
>     crashed.
> 
>     Where can I replace '-' by '_' in the soaplab2 sources ?
> 
> 
> The name of the directory in question is also the job ID. It is created 
> in org.soaplab.services.JobManager in method createJobId(). The minus is 
> given by calling 'new java.rmi.server.UID().toString()'.
> 
> Martin

So, it should work if I use 
java.rmi.server.UID().toString().replaceAll('-', '_')
but I need to recompile this class and reinstall soaplab2 ?

I don't know yet if the bug is coming from the application, or from 
pascal syntax !


Could you add this replaceAll("-", "_") method to soaplab2 source code 
to avoid this kind of problem in the future for other users & maybe for 
other old pascal programs ?

Thanks

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

>>     More precisely, this is '-', in the path, which makes my program
>>     crashed.
>>
>>     Where can I replace '-' by '_' in the soaplab2 sources ?
>>
>>
>> The name of the directory in question is also the job ID. It is 
>> created in org.soaplab.services.JobManager in method createJobId(). 
>> The minus is given by calling 'new java.rmi.server.UID().toString()'.
>>
>> Martin
> 
> So, it should work if I use 
> java.rmi.server.UID().toString().replaceAll('-', '_')
> but I need to recompile this class and reinstall soaplab2 ?
> 
> I don't know yet if the bug is coming from the application, or from 
> pascal syntax !
> 
> 
> Could you add this replaceAll("-", "_") method to soaplab2 source code 
> to avoid this kind of problem in the future for other users & maybe for 
> other old pascal programs ?
> 
> Thanks

It works perfectly with .replaceAll("-", "_")

So, could you add this replaceAll("-", "_") method to soaplab2 source 
code to avoid this kind of problem in the future for other users & maybe 
with other old pascal programs ?

Thanks

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

> So, could you add this replaceAll("-", "_") method to soaplab2 source
> code to avoid this kind of problem in the future for other users & maybe
> with other old pascal programs ?


I am aware that you are waiting for my answer to this. But I hesitate...
Next time, it may be another character, next time another... I probably
still feel that this should be dealt with where it belongs - meaning in your
application, or in wrapping your application. The minus sign is a valid
character in a filename in all major platforms (unix, windows, mac) - at
least I think so.

Mahmut, what is your opinion?

Cheers,
Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Mahmut U. <ul...@eb...>]

Hi Martin,

>         So, could you add this replaceAll("-", "_") method to soaplab2
>         source
>         code to avoid this kind of problem in the future for other
>         users & maybe
>         
>         with other old pascal programs ?
> 
> I am aware that you are waiting for my answer to this. But I
> hesitate... Next time, it may be another character, next time
> another... I probably still feel that this should be dealt with where
> it belongs - meaning in your application, or in wrapping your
> application. The minus sign is a valid character in a filename in all
> major platforms (unix, windows, mac) - at least I think so.
> 
> Mahmut, what is your opinion?

I have noticed that the minus signs related to the numeric value of two
attributes in the UID machinery. Although it would not look pretty it
should not harm the uniqueness of the identifiers if we replace minus
sign characters by some other character. 

Since what we do here is in some sense similar to what Java
File.createTempFile does in getting unique file identifiers, I checked
its source code. Their unique file identifier is based on a static
integer counter which is initialised by a "positive" random number,
however, after the maximum integer value reached the counter does have
minus values and so the unique file names have minus sign.

Regards,
Mahmut

Re: [Soaplab-users] Problem with Infile Path

[Martin S. <mar...@gm...>]

Okay, let's do the replacement, minus for underscore.
Thanks, Mahmut.

Martin

-- 
Martin Senger
email: mar...@gm...,m.s...@cg...
skype: martinsenger

Re: [Soaplab-users] Problem with Infile Path

[Sebastien M. <Seb...@un...>]

> Okay, let's do the replacement, minus for underscore.
> Thanks, Mahmut.
> 
> Martin

Okay,
I will investigate more deeply to find if it is only pascal code dependent.

Thanks

-- 
Sébastien Moretti
SIB, Quartier Sorge - Genopode
CH-1015 Lausanne, Switzerland
Tel.: +41 (21) 692 4056/4221