[staf-users] Fw: Stack Overflow ??

[Sharon L. <lu...@us...>]

Actually in re-reading your note, I think you were saying that the high 
CPU usage was occurring when the Python code you wrote (not STAF/STAX 
code) is does something with a process's large stdout file.   If so, not 
enough information was provided.  I don't know what you are trying to to 
do.  Only snippets of code was provided, etc.  This is probably more of a 
Python coding question than a STAF/STAX question.

--------------------------------------------------------------
Sharon Lucas
IBM Austin,   lu...@us...
(512) 286-7313 or Tieline 363-7313

----- Forwarded by Sharon Lucas/Austin/IBM on 01/29/2010 11:06 AM -----

Sharon Lucas/Austin/IBM@IBMUS 
01/29/2010 10:34 AM

To
anas ameziane <ana...@gm...>
cc
sta...@li...
Subject
Re: [staf-users] Stack Overflow ??







First, you need to understand that if you return a file using the 
RETURNSTDOUT (or RETURNSTDERR/RETURNFILE) options on a PROCESS START 
request (or a <process> element in a STAX job), the contents of the file 
are read and then sent as a string string.  If the file is large, this can 
take up a lot of memory and can consure a lot of CPU by the unmarshall 
process.  Sub-section "8.12,2 Start" that talks about the PROCESS service 
in the STAF User's Guide at 
http://staf.sourceforge.net/current/STAFUG.htm#HDRPROCSRV contains the 
following information that recommends that you don't use the RETURNxxx 
options to return large files and talks about how you can specify a 
maximum size for a returned file:

Since the entire contents of returned files are stored in the result 
string, if you attempt to return the contents of a very large file, you 
may run out of memory so it is not recommended that you use the 
RETURNSTDOUT, RETURNSTDERR, or RETURNFILE options to return large files. 
To help prevent this problem, you can specify a maximum size for a file 
returned by this request by setting the MAXRETURNFILESIZE operational 
parameter in the STAF configuration file on the machine where the process 
is run, or by setting the STAF/MaxReturnFileSize variable in the request 
variable pool of the handle that submitted the request. The lowest of 
these two values is used as the maximum return file size (not including 0 
which indicates no limit). 

So, for large files created by processes, instead of returning their 
contents this way, you may want to use the FS COPY request to copy the 
contents to the STAX service machine after the process completes (if you 
really need the file to reside on the STAX service machine) as the FS COPY 
request copies the files in "chunks" so it doesn't take up much 
CPU/memory.  Then you could read the local file using Python via a 
<script> element if you needed to.  Or, it might be more efficient if you 
can leave the process stdout file on the remote machine and if you simply 
need to check the contents of the file for some information, you couldrun 
another process on the remote machine that does this. 

--------------------------------------------------------------
Sharon Lucas
IBM Austin,   lu...@us...
(512) 286-7313 or Tieline 363-7313



anas ameziane <ana...@gm...> 
01/29/2010 08:50 AM 


To
sta...@li... 
cc

Subject
[staf-users] Stack Overflow ??








Hello to all

I use STAF / STAX to automate some tasks.
I developed a python class that allows me to set the stdout of each 
process node in XML.

the problem that it works well, except when stdout is large (7 mega 
octects).
java.exe consumes 99.9% of my CPU, and I see no error in the logs (and 
staf jvm).

----------------

<script>
    proc = Process(procName)
</script>
...
<process name="'copygrep'">
    <location>location</location>
    <command mode="'shell'">'%s/copygrep.sh'%(dir)</command>
    <parms>'%s'%(params)</parms>
    <console use="'same'"/> 
    <stderr mode="'stdout'"/>
  <returnstdout/>
</process>
 
<script>
    proc._end([RC,STAXResult[0][1]])
</script>
...

-----------------------------------------------------

<script>
    class Process:
        def __init__(self,name):
            self.name = name
            self.stdout = 'NaN'
            self.rc ='-1'
 
    ...
 
        def _end(self,resultTab):
            if resultTab[0] != '' :
                self.rc = str(resultTab[0])
            if resultTab[1] != '' :
                import re
                self.stdout = resultTab[1]
    ...
 
        def getXMLResult(self):
            from xml.dom.minidom import Document
            xmlDoc = Document()
            ...
            stdout = xmlDoc.createElement("stdout")
            stdout.appendChild(xmlDoc.createTextNode(self.stdout))
            process.appendChild(stdout)
    ...
 
</script>

it is an overflow , a bug or something else ?
any ideas ?

Anas A.

Thanks,
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the 
business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com_______________________________________________

staf-users mailing list
sta...@li...
https://lists.sourceforge.net/lists/listinfo/staf-users
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the 
business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
staf-users mailing list
sta...@li...
https://lists.sourceforge.net/lists/listinfo/staf-users