Abnormal memory consuming ?

[T. Delecolle]

I'm using the MxQuery 0.6.0 to produce xml and after analysis with NetBeans, i am quite surprised by the memory used by my java application.

For example, my application uses about 600 MB in the JVM when applying the query below to the following input file.

- Input File 1.xml (100 couple <NODE_12/><NODE_13/>):

<NODE_1>
<NODE_11 BID="1" BSIDE="1" LD_KEY="3456" LCODE="12" NAME="MODULE_1"/>

<NODE_12 ATTR_IO_1="0000" ATTR_IO_2="AA" ATTR_IO_3="AB" ATTR_IO_4="AC" ATTR_KEY_1="123" ATTR_KEY_2="234" ATTR_KEY_3="345" ATTR_KEY_4="456" ATTR_FAIL="0" CODE="0" FCODE="0" FTYPE="TXT1" FECODE="0" FIRST_ARP="0.0" FIRST_IO="Text5" FIRST_LCODE="10" FOURTH_IO="Text2" FOURTH_LCODE="40" ATTR_LR="0" O_1_1="0" O_1_2="12" O_2_1="21" O_2_2="22" O_3_1="31" O_3_2="32" O_4_1="41" O_4_2="42" PFIN_1="1" PFIN_2="2" PFIN_3="3" PFIN_4="4" PRIORITY="Low" SECOND_ARP="0.0" SECOND_IO="Text3" SECOND_LCODE="20" SL_1="Text8" SL_2="Text8" SL_3="Text8" SL_4="Text8" THIRD_ARP="0.0" THIRD_IO="Text7" THIRD_LCODE="30" MODE="Specific "/>
<NODE_13 CODE="0" ATTR_ID="0"/>

<NODE_12 ATTR_IO_1="0000" ATTR_IO_2="AA" ATTR_IO_3="AB" ATTR_IO_4="AC" ATTR_KEY_1="123" ATTR_KEY_2="234" ATTR_KEY_3="345" ATTR_KEY_4="456" ATTR_FAIL="1" CODE="1" FCODE="1" FTYPE="TXT1" FECODE="1" FIRST_ARP="0.1" FIRST_IO="Text3" FIRST_LCODE="10" FOURTH_IO="Text2" FOURTH_LCODE="40" ATTR_LR="0" O_1_1="1" O_1_2="12" O_2_1="21" O_2_2="22" O_3_1="31" O_3_2="32" O_4_1="41" O_4_2="42" PFIN_1="1" PFIN_2="2" PFIN_3="3" PFIN_4="4" PRIORITY="Low" SECOND_ARP="0.0" SECOND_IO="Text3" SECOND_LCODE="20" SL_1="Text8" SL_2="Text8" SL_3="Text8" SL_4="Text8" THIRD_ARP="0.0" THIRD_IO="Text7" THIRD_LCODE="30" MODE="Specific "/>
<NODE_13 CODE="1" ATTR_ID="1"/>

...

<NODE_12 ATTR_IO_1="0000" ATTR_IO_2="AA" ATTR_IO_3="AB" ATTR_IO_4="AC" ATTR_KEY_1="123" ATTR_KEY_2="234" ATTR_KEY_3="345" ATTR_KEY_4="456" ATTR_FAIL="99" CODE="99" FCODE="4" FTYPE="TXT1" FECODE="99" FIRST_ARP="0.9" FIRST_IO="Text4" FIRST_LCODE="10" FOURTH_IO="Text2" FOURTH_LCODE="40" ATTR_LR="0" O_1_1="3" O_1_2="12" O_2_1="21" O_2_2="22" O_3_1="31" O_3_2="32" O_4_1="41" O_4_2="42" PFIN_1="1" PFIN_2="2" PFIN_3="3" PFIN_4="4" PRIORITY="Low" SECOND_ARP="0.9" SECOND_IO="Text3" SECOND_LCODE="20" SL_1="Text8" SL_2="Text8" SL_3="Text8" SL_4="Text8" THIRD_ARP="0.0" THIRD_IO="Text7" THIRD_LCODE="30" MODE="Specific "/>
<NODE_13 CODE="99" ATTR_ID="99"/>

</NODE_1>

- Request performed :

declare namespace pct_comp="java:cmComputations.general.PCTComputations";

declare function local:getFSI () {
    1
};

declare function local:getFSL ($bdd_1) {
    $bdd_1/NODE_11/@BSIDE
};

declare function local:getFI ($bdd_13) {
    $bdd_13/@ATTR_ID
};

declare function local:getP ($bdd_12) {
    if ($bdd_12/@PRIORITY = "Low") then
        <FIELD_INFO INDEX="0" TYPE="" NAME="PRIORITY" VALUE="0" COMMENT="PRIORITY" />
    else
        if ($bdd_12/@PRIORITY = "High") then
            <FIELD_INFO INDEX="0" TYPE="" NAME="PRIORITY" VALUE="1" COMMENT="PRIORITY" />
        else
            <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-1')}"/>
};

declare function local:getMode ($bdd_12) {
    if ($bdd_12/@MODE = "Periodic ") then
        <FIELD_INFO INDEX="1" TYPE="" NAME="MODE" VALUE="0" COMMENT="MODE" />
    else if ($bdd_12/@MODE = "Specific ") then
        <FIELD_INFO INDEX="1" TYPE="" NAME="MODE" VALUE="1" COMMENT="MODE" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-2')}"/>
};

declare function local:getThirdARP ($bdd_12) {
    if ( fn:string-length($bdd_12/@THIRD_ARP) < 1 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="0" COMMENT="THIRD_ARP" >
                ( { java:raiseInfo ("default value 0")} )
        </FIELD_INFO>
    else if ( $bdd_12/@THIRD_ARP = 0.0) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="0" COMMENT="THIRD_ARP" />
    else if ( 0 < $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.1 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="1" COMMENT="THIRD_ARP" />
    else if ( 0.1 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.2 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="2" COMMENT="THIRD_ARP" />
    else if ( 0.2 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.3 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="3" COMMENT="THIRD_ARP" />
    else if ( 0.3 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.4 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="4" COMMENT="THIRD_ARP" />
    else if ( 0.4 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.5 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="5" COMMENT="THIRD_ARP" />
    else if ( 0.5 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.6 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="6" COMMENT="THIRD_ARP" />
    else if ( 0.6 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.7 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="7" COMMENT="THIRD_ARP" />
    else if ( 0.7 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.8 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="8" COMMENT="THIRD_ARP" />
    else if ( 0.8 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 0.9 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="9" COMMENT="THIRD_ARP" />
    else if ( 0.9 <= $bdd_12/@THIRD_ARP and $bdd_12/@THIRD_ARP < 1.0) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="10" COMMENT="THIRD_ARP" />
    else if ( $bdd_12/@THIRD_ARP = 1.0 ) then
        <FIELD_INFO INDEX="2" TYPE="" NAME="THIRD_ARP" VALUE="11" COMMENT="THIRD_ARP" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-3')}"/>
};

declare function local:getLink23 ($bdd_12) {
    if (fn:string-length($bdd_12/@ATTR_LR) < 1 ) then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" >
                ( { java:raiseInfo ("default value 0")} )
        </FIELD_INFO>
    else if ( $bdd_12/@ATTR_LR = "0") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A or B") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A and B") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A or (B and C)") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A or B or C") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="1" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "(A and B) or C") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="1" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A and (B or C)") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="1" COMMENT="LINK23" />
    else if ($bdd_12/@ATTR_LR = "A and B and C") then
        <FIELD_INFO INDEX="3" TYPE="" NAME="LINK23" VALUE="0" COMMENT="LINK23" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-4')}"/>
};

declare function local:getSecondARP ($bdd_12) {
    if (fn:string-length($bdd_12/@SECOND_ARP) < 1 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="0" COMMENT="SECOND_ARP" >
                ( { java:raiseInfo ("default value 0")} )
        </FIELD_INFO>
    else if ( $bdd_12/@SECOND_ARP = 0) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="0" COMMENT="SECOND_ARP" />
    else if ( 0 < $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.1 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="1" COMMENT="SECOND_ARP" />
    else if ( 0.1 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.2 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="2" COMMENT="SECOND_ARP" />
    else if ( 0.2 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.3 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="3" COMMENT="SECOND_ARP" />
    else if ( 0.3 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.4 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="4" COMMENT="SECOND_ARP" />
    else if ( 0.4 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.5 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="5" COMMENT="SECOND_ARP" />
    else if ( 0.5 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.6 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="6" COMMENT="SECOND_ARP" />
    else if ( 0.6 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.7 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="7" COMMENT="SECOND_ARP" />
    else if ( 0.7 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.8 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="8" COMMENT="SECOND_ARP" />
    else if ( 0.8 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 0.9 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="9" COMMENT="SECOND_ARP" />
    else if ( 0.9 <= $bdd_12/@SECOND_ARP and $bdd_12/@SECOND_ARP < 1.0) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="10" COMMENT="SECOND_ARP" />
    else if ( $bdd_12/@SECOND_ARP = 1.0 ) then
        <FIELD_INFO INDEX="4" TYPE="" NAME="SECOND_ARP" VALUE="11" COMMENT="SECOND_ARP" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-5')}"/>
};

declare function local:getLink12 ($bdd_12) {
    if (fn:string-length($bdd_12/@ATTR_LR) < 1 ) then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" >
                ( { java:raiseInfo ("default value 0")} )
        </FIELD_INFO>
    else if ( $bdd_12/@ATTR_LR = "0") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A or B") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="1" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A and B") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A or (B and C)") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="1" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A or B or C") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="1" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "(A and B) or C") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A and (B or C)") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else if ($bdd_12/@ATTR_LR = "A and B and C") then
        <FIELD_INFO INDEX="5" TYPE="" NAME="LINK12" VALUE="0" COMMENT="LINK12" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-6')}"/>
};

declare function local:getFirstARP ($bdd_12) {
    if (fn:string-length($bdd_12/@FIRST_ARP) < 1 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="0" COMMENT="FIRST_ARP" >
                ( { java:raiseInfo ("default value 0")} )
        </FIELD_INFO>
    else if ( $bdd_12/@FIRST_ARP = 0) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="0" COMMENT="FIRST_ARP" />
    else if ( 0 < $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.1 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="1" COMMENT="FIRST_ARP" />
    else if ( 0.1 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.2 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="2" COMMENT="FIRST_ARP" />
    else if ( 0.2 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.3 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="3" COMMENT="FIRST_ARP" />
    else if ( 0.3 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.4 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="4" COMMENT="FIRST_ARP" />
    else if ( 0.4 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.5 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="5" COMMENT="FIRST_ARP" />
    else if ( 0.5 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.6 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="6" COMMENT="FIRST_ARP" />
    else if ( 0.6 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.7 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="7" COMMENT="FIRST_ARP" />
    else if ( 0.7 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.8 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="8" COMMENT="FIRST_ARP" />
    else if ( 0.8 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 0.9 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="9" COMMENT="FIRST_ARP" />
    else if ( 0.9 <= $bdd_12/@FIRST_ARP and $bdd_12/@FIRST_ARP < 1.0) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="10" COMMENT="FIRST_ARP" />
    else if ( $bdd_12/@FIRST_ARP = 1.0 ) then
        <FIELD_INFO INDEX="6" TYPE="" NAME="FIRST_ARP" VALUE="11" COMMENT="FIRST_ARP" />
    else
        <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-7')}"/>
};

declare function local:getSFMD ($bdd_12) {
    <STRUCT TYPE="const T_FMD" INDEX="3" NAME="FMD" COMMENT="">
        { local:getP($bdd_12) }
        { local:getMode($bdd_12) }
        { local:getThirdARP($bdd_12) }
        { local:getLink23($bdd_12) }       
        { local:getSecondARP($bdd_12) }
        { local:getLink12($bdd_12) }       
        { local:getFirstARP($bdd_12) }
        <FIELD_INFO INDEX="7" TYPE="" NAME="ATTR_FAIL" VALUE="{$bdd_12/@ATTR_FAIL}" COMMENT="ATTR_FAIL" />
        <FIELD_INFO INDEX="8" TYPE="" NAME="CODE" VALUE="{$bdd_12/@CODE}" COMMENT="CODE" />
        <FIELD_INFO INDEX="9" TYPE="" NAME="NOT_USED_FIELD_1" VALUE="0" COMMENT="NOT_USED_FIELD_1" />
    </STRUCT>
};

declare function local:getISFTRank () {
    <FIELD_INFO INDEX="6" TYPE="" NAME="A_ISFT_RANK" VALUE="0" COMMENT="A_ISFT_RANK" />
};

declare function local:processMD () {
    <FAILURE_MESSAGES>
    {
    let $bdd_1 := doc('BDD')/NODE_1
    return
        for $bdd_13 in $bdd_1/NODE_13
        let $bdd_12 :=  $bdd_1/NODE_12[@CODE = $bdd_13/@CODE]
            return
                if (count($bdd_12) = 1) then
                    let $FSI := local:getFSI()
                    let $FSL := local:getFSL($bdd_1)
                    let $FI := local:getFI($bdd_13)
                    return
                        let $IFSI := java:computeInternalFailureIdentifier ( string($FSI),
                                                                                     string($FSL),
                                                                                     string($FI))
                        return
                        <STRUCT TYPE="const T_FMSFABD0100Record" INDEX="0" NAME="" COMMENT="Element 0">
                       
                            <FIELD_INFO INDEX="0" TYPE="" NAME="FSI" VALUE="{ $FSI }" COMMENT="FSI" />
                            <FIELD_INFO INDEX="1" TYPE="" NAME="FSL" VALUE="{ $FSL }" COMMENT="FSL" />
                            <FIELD_INFO INDEX="2" TYPE="" NAME="FI" VALUE="{ $FI }" COMMENT="FI" />
                            { local:getSFMD($bdd_12) }
                            <FIELD_INFO INDEX="4" TYPE="" NAME="A_LSFT_RANK" VALUE="TBD" COMMENT="A_LSFT_RANK" />
                            <FIELD_INFO INDEX="5" TYPE="" NAME="A_ISFT_RANK" VALUE="TBD" COMMENT="A_ISFT_RANK" />
                            { local:getISFTRank() }
                            <FIELD_INFO INDEX="7" TYPE="" NAME="SRU_SUSPECTED" VALUE="0" COMMENT="SRU_SUSPECTED" />
                            <FIELD_INFO INDEX="8" TYPE="" NAME="FDE_NUMBER_OF_ELEMENTS" VALUE="1" COMMENT="FDE_NUMBER_OF_ELEMENTS" />
                            <FIELD_INFO INDEX="9" TYPE="" NAME="A_FDETable_RANK" VALUE="TBD" COMMENT="A_FDETable_RANK" />
                            <FIELD_INFO INDEX="10" TYPE="" NAME="NOT_USED_FIELD_2" VALUE="0" COMMENT="NOT_USED_FIELD_2" />
                            <FIELD_INFO INDEX="11" TYPE="" NAME="_CODE" VALUE="{$bdd_13/@CODE}" COMMENT="_CODE" />
                            <FIELD_INFO INDEX="12" TYPE="" NAME="_INTERNAL_FAILURE_IDENTIFIER" VALUE="{ $IFSI }" COMMENT="_INTERNAL_FAILURE_IDENTIFIER" />
                        </STRUCT>
                else
                    <ERROR ERROR_NUMBER="{ java:raiseError ('ERR-8', 'Impossible to link nodes /NODE_1/NODE_13 and /NODE_1/NODE_12/ according to CODE attribute for LRU <POSITION_ID>')}"/>
    }
    </FAILURE_MESSAGES>
};

declare function local:orderAll ($processed_FMSF) {
    for $struct in $processed_FMSF/STRUCT
    order by $struct/FIELD_INFO[ @NAME="_INTERNAL_FAILURE_IDENTIFIER"]/@VALUE
    return
        $struct
};   

copy $cm := doc('outputFileName')/CODE_MODEL

modify (
    insert node
        <ARRAY INDEX="1" TYPE="const T_FMSFABD0100Record" NAME="FMSFABD0100Record_<POSITION_ID>" COMMENT="FAILURE MESSAGE TABLE" BRACKET_USED="YES" IN_LINE="NO">
            { local:orderAll(local:processMD ()) }
        </ARRAY>
    as last into $cm/FILE[ @NAME = 'FailureMessageTables_<POSITION_ID>_CFG' ]
)

return $cm

I wonder how to optimize this as i need to manipulate a larger input file (with about 4000 couples instead of the 100 used in first file).
At the moment, my java application (which have about 10 queries with the same complexity than the one above) exits with a OutOfMemory Error since i have more than 100 couples in my input file (my JVM can use 1GB of memory).

Do i have to split my query in several ones ?

PS : i am sorry for the "english mistakes"...

[Peter M. Fischer]

Hi,

the code looks reasonably OK.

We are trying to reproduce how this memory consumption occurs. Is there any chance that we can have a larger input file (e.g. your 100 pairs file) and code for the missing functions (java:*)?

How do you call this XQuery code? On the command line, using XQJ or the MXQuery Java API?

If you already did some analysis in Netbeans (I assume profiling), is there anything that stands out?

Best Regards,

Peter

(MXQuery developer)

[T. Delecolle]

I call this XQuery code by using the MxQuery Java API : this code is part of a java application.

For the missing functions (java:*), I have declared them in a file EFG.xml (External Function Gallery) with that format :
<functionGallery>
    <functionDescription>
        <functionPrefix>java</functionPrefix>
        <functionName>raiseError</functionName>
        <arity>2</arity>
        <className>externalFG.A350.general.RaiseError</className>
    </functionDescription>
    <functionDescription>
        <functionPrefix>java</functionPrefix>
        <functionName>raiseInfo</functionName>
        <arity>1</arity>
        <className>externalFG.A350.general.RaiseInfo</className>
    </functionDescription>
    <functionDescription>
        <functionPrefix>java</functionPrefix>
        <functionName>computeInternalFailureIdentifier</functionName>
        <arity>3</arity>
        <className>externalFG.A350.general.ComputeInternalFailureIdentifier</className>
    </functionDescription>
</functionGallery>

and I have added parsing of EFG.xml in function createStdFunctionGallery in file ch.ethz.mxquery.functions.FunctionGallery.java

Functions used (raiseError and raiseInfo) allow simply to print a message to the standard output (these functions work without any problem and the problem of memory occurs even if they are not used).

I've built the 100 pairs file. I just hope that it works (i have changed every node name and every value to hide the real work...) fine with the previous posted query. Do you want me to copy/paste it in the thread or do you want to have the complete file by mail ?

From the Netbeans analysis (done with a little configuration), I see that 60% of my processor computing time is used for parsing the queries and more than 38% is used for executing the query.
The memory profiling shows that consumption comes from String (at 11%) / char [] (at 20%) / byte[] (at 17%).
The String and char[] allocations go through the Garbage Collector : they live for about 100 generations (more than all other objects) and take a lot of memory.

We try to split the large queries in many small ones and we have seen a positive impact on performance...
Is there any other known practices to improve the global performance (at parsing or at executing) ?

Regards,

Thomas Delécolle.

[Peter M. Fischer]

Hello again,

regarding the input file - I am fine with either way. If you want to send it by mail, either use the address listed on the MXQuery web site or send it directly to my ETH Zurich address (peter.fischer at inf.ethz.ch)

Best Regards,
Peter Fischer

[Soh Guan Hoe]

Hi I'm new to MXQuery so was trying out the examples in the Download. Everything was smooth until I came to a simple example of when my input XML file is very huge (e.g 113MB), the OutOfMemory error is raised. When I reduce the file size to say 1 MB it works again.

I believe MXQuery would have some API which I can use to indicate it should do streaming and processing for huge XML document cuz I do not think it intend to load the huge XML file into memory and process isn't it ?

May I know which API call to activate MXQuery to do streaming and processing to avoid that OutOfMemory error ?

Thanks.

[Peter M. Fischer]

Dear sohguanh,

could you please provide some more details? MXQuery tries to do streaming whenever possible, but there are certain factors that might prohibit its usage:
- Update and Fulltext require complete materialization, the first to perform changes, the second for indexing. In MXQuery 0.6.0, enabling these features would trigger materialization for all data. In the upcoming release, the query is analyzed, and materialization is only activated on those sources which actually need it
- Certain query expressions need to materialize intermediate data, e.g. if you do a nested-loop join. Without knowing what you are doing, it is hard to tell what the problem is

Best Regards,
Peter Fischer