#1390 Hang when getting content of a file on linux

Unix::Linux
open
nobody
5
2014-08-19
2011-02-25
No

David Bender has the file in question already, this exact file works fine on Windows.

staf local stax execute file /root/gcs/temp/teststax.xml wait returnresult

STAX code (teststax.xml) to reproduce:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE stax SYSTEM "stax.dtd">
<stax>
<defaultcall function="main"/>
<function name="main">
<sequence>
<stafcmd>
<location>'local'</location>
<service>'FS'</service>
<request>'GET FILE /root/gcs/temp/services TEXT FORMAT ASIS'</request>
</stafcmd>
<message log="1">'RC=%s, Result=%s' % (RC, STAFResult)</message>
<return>STAFResult</return>
</sequence>
</function>
</stax>

Discussion

  • Note that the result is the same without the TEXT FORMAT ASIS:

    <request>'GET FILE /tmp/services'</request>

    The STAX job log shows the following:

    20110225-15:33:41 Start JobID: 5, File: /tmp/getTextFile.xml, Machine: local:/
    /local, Function: , Args: , JobName: <N/A>
    20110225-15:33:42 Error STAXJob$STAFQueueMonitor.run(): Exception unmarshallin
    g queued messages. Marshalled string: @SDT/*:673701:@
    SDT/{:761::13:map-class-map@SDT/{:733::24:STAF/Service
    /Queue/Entry@SDT/{:694::4:keys@SDT/[8:633:@SDT/{:91::1
    2:display-name@SDT/$S:8:Priority:18:display-short-name
    @SDT/$S:1:P:3:key@SDT/$S:8:priority@SDT/{:60::12:displ
    ay-name@SDT/$S:9:Date-Time:3:key@SDT/$S:9:timestamp@SD
    T/{:56::12:display-name@SDT/$S:7:Machine:3:key@SDT/$S:
    7:machine@SDT/{:101::12:display-name@SDT/$S:11:Handle
    Name:18:display-short-name@SDT/$S:4:Name:3:key@SDT/$S:
    10:handleName@SDT/{:88::12:display-name@SDT/$S:6:Handl
    e:18:display-short-name@SDT/$S:2:H#:3:key@SDT/$S:6:han
    dle@SDT/{:50::12:display-name@SDT/$S:4:User:3:key@SDT/
    $S:4:user@SDT/{:50::12:display-name@SDT/$S:4:Type:3:ke
    y@SDT/$S:4:type@SDT/{:56::12:display-name@SDT/$S:7:Mes
    sage:3:key@SDT/$S:7:message:4:name@SDT/$S:24:STAF/Serv
    ice/Queue/Entry@SDT/[1:672914:@SDT/%:672900::24:STAF/S
    ervice/Queue/Entry@SDT/$S:1:5@SDT/$S:17:20110225-15:33
    :42@SDT/$S:13:local://local@SDT/$S:12:STAF_Process@SDT
    (More...)

     
  • Sharon Lucas
    Sharon Lucas
    2011-02-28

    I narrowed down the line (actually the character) in the services file that is causing the problem. It is line #2449 that contains character é (E WITH ACUTE') , \u00E9 Unicode character:

    # Stéphane MONS <st.mons@wanadoo.fr> September 2005

    If you change the é character to a regular lower-case e character, then this problem does not occur.

    This is a multi-byte character, so there may be an issue in handling multi-byte characters in the STAF Jython marshalling methods. I'm investigating further.

    # staf -verbose local log query machine {STAF/Config/MachineNickname} LOGNAME STAX_Job_56
    Response
    --------
    [
    {
    Date-Time: 20110228-15:48:38
    Level : Start
    Message : JobID: 56, File: /tmp/sharon/getTextFileOrg.xml, Machine: local:/
    /local, Function: , Args: , JobName: <N/A>
    }
    {
    Date-Time: 20110228-15:48:38
    Level : Error
    Message : STAXJob$STAFQueueMonitor.run(): Exception unmarshalling queued me
    ssages.
    Marshalled string: @SDT/*:1145:@SDT/{:761::13:map-class-map@SDT/{:733::24:STAF/Service/Queue/Entry@SDT/{:694::4:keys@SDT/[8:633:@SDT/{:91::12:display-name@SDT/$S:8:Priority:18:display-short-name@SDT/$S:1:P:3:key@SDT/$S:8:priority@SDT/{:60::12:display-name@SDT/$S:9:Date-Time:3:key@SDT/$S:9:timestamp@SDT/{:56::12:display-name@SDT/$S:7:Machine:3:key@SDT/$S:7:machine@SDT/{:101::12:display-name@SDT/$S:11:Handle Name:18:display-short-name@SDT/$S:4:Name:3:key@SDT/$S:10:handleName@SDT/{:88::12:display-name@SDT/$S:6:Handle:18:display-short-name@SDT/$S:2:H#:3:key@SDT/$S:6:handle@SDT/{:50::12:display-name@SDT/$S:4:User:3:key@SDT/$S:4:user@SDT/{:50::12:display-name@SDT/$S:4:Type:3:key@SDT/$S:4:type@SDT/{:56::12:display-name@SDT/$S:7:Message:3:key@SDT/$S:7:message:4:name@SDT/$S:24:STAF/Service/Queue/Entry@SDT/[1:361:@SDT/%:350::24:STAF/Service/Queue/Entry@SDT/$S:1:5@SDT/$S:17:20110228-15:48:38@SDT/$S:13:local://local@SDT/$S:12:STAF_Process@SDT/$S:1:1@SDT/$S:16:none://anonymous@SDT/$S:20:STAF/RequestComplete@SDT/$S:155:@SDT/{:144::2:rc@SDT/$S:1:0:13:requestNumber@SDT/$S:6:112161:6:result@SDT/$S:75:#
    Stéphane MONS <st.mons@wanadoo.fr> September 2005

    java.lang.String incompatible with java.util.List
    java.lang.ClassCastException: java.lang.String incompatible with java.util.List
    at com.ibm.staf.service.stax.STAXJob$STAFQueueMonitor.run(STAXJob.java:2931)
    }
    ]

     
  • Sharon Lucas
    Sharon Lucas
    2011-03-01

    The problem isn't in STAF Jython marshallihg methods. The length of the string containing this character is incorrect. Most likely the problem has to do with the STAF codepage conversion code as the problem occurs on Linux when STAF uses the UTF-8 codepage but the problem does not occur on Windows (where a different codepage is used).
    multi-byte characters in the STAF Jython marshalling methods. I'm
    investigating further

     
  • Sharon Lucas
    Sharon Lucas
    2012-04-30

    I wrote the following C++ program (listdir.cpp) to see what STAF thinks the length of Stéphane is (in characters as well as in bytes):

    #include "STAFOSTypes.h"
    #include "STAFFileSystem.h"

    int main(int argc, char **argv)
    {
    STAFHandlePtr handlePtr;

    unsigned int rc = STAFHandle::create("Test", handlePtr);

    if (rc != 0)
    {
    cout << "Error registering with STAF, RC: " << rc << endl;
    return rc;
    }

    STAFString machine = "local";
    STAFString service = "FS";
    STAFString request = "GET FILE /tmp/badChar.txt";

    cout << "STAF local FS GET FILE /tmp/badChar.txt" << endl;
    STAFResultPtr result = handlePtr->submit(machine, service, request);

    cout << "RC: " << result->rc << ", Result: " << result->result << endl;

    cout << "Length (kChar) : " << result->result.length(STAFString::kChar) << endl;
    cout << "Length (kBytes): " << result->result.length() << endl;

    STAFString newResult = handlePtr->wrapData(result->result);

    cout << "Wrapped Result: " << newResult << endl;

    return 0;
    }

    On my Windows laptop, when I run this C++ program, it returns 8 for the length in characters and 9 for the length in bytes:

    C:\tmp>listdir
    STAF local FS GET FILE /tmp/badChar.txt
    RC: 0, Result: StΘphane

    Length (kChar) : 8
    Length (kBytes): 9
    Wrapped Result: :8:StΘphane

    On Linux (staf1e), when I run this C++ program, it returns 7 for the length in characters and 9 for the length in bytes:

    [root@staf1e staf]# listdir
    STAF local FS GET FILE /tmp/badChar.txt
    RC: 0, Result: St▒phane

    Length (kChar) : 7
    Length (kBytes): 9
    Wrapped Result: :7:St▒phane

    So, the STAFString length() method is returning the wrong length in characters (7 instead of 8) for Stéphane. Need to investigate why.

     
  • Sharon Lucas
    Sharon Lucas
    2012-05-01

    As a workardound for this issue, you can change STAF to not use the UTF-8 codepage and instead use the LATIN-1 codepage which supports the é character. You can do this by setting the STAFCODEPAGEOVERRIDE environment variable and then re-start STAFProc.

    # STAF local SHUTDOWN SHUTDOWN
    # export STAFCODEPAGEOVERRIDE=LATIN-1
    # cd /usr/local/staf
    # ./startSTAFProc.sh

    Verify that STAFProc is now using the LATIN-1 ocdepage:

    # STAF local VAR RESOLVE STRING {STAF/Config/CodePage}
    Response
    --------
    LATIN-1

    The problem then no longer occurs when a STAX job submits a FS GET FILE request on this system to access a file that contains the é character.