[Servingxml-help] out of memory issues

[Ken R. <Ken...@en...>]

Hello,
I'm new here so please be gentle.

I've been tasked with mapping a large (2gig) flat file to xml and servicexml looks like the route to take to do that mapping.

The file is of the format:

BEG|id1|f3|f4|f5
DETBEG|id1|f2|f3
DET|id1|f2|f3
DET|id1|f2|f3
DET|id1|f2|f3
DET|id1|f2|f3
DETEND|id1|f2|f3
END|id1|f2
BEG|id2|f2|f3|f4
DETBEG|id2|f2|f3
DET|id2|f2|f3
DET|id2|f2|f3
DET|id2|f2|f3
DET|id2|f2|f3
DETEND|id2|f2|f3
END|id2|f2

Each record is surrounded by the BEG/END lines

To start with I have been trying to map without the details to get the hang of how servingxml works.

I have the following resource.xml:

<?xml version="1.0"?>
<sx:resources xmlns:sx="http://www.servingxml.com/core">
  <sx:service id="myData">
            <sx:serialize>
            <sx:xsltSerializer>
                        <sx:outputProperty name="indent" value="yes"/>
                  </sx:xsltSerializer>
              <sx:transform>
                        <sx:content ref="myData"/>
              </sx:transform>
            </sx:serialize>
  </sx:service>

  <sx:recordContent id="myData">
            <sx:flatFileReader>
              <sx:fileSource file="data/test.flat"/>
              <sx:flatFile ref="myFlatFile"/>
            </sx:flatFileReader>
            <sx:recordMapping ref="toXmlMapping"/>
  </sx:recordContent>

  <sx:flatFile id="myFlatFile">
            <sx:flatFileBody>
              <sx:flatRecordTypeChoice>
                        <sx:fieldDelimiter value="|"/>
                        <sx:delimitedField name="recordType"/>
                        <sx:delimitedField name="id"/>
                        <sx:when test="recordType='BEG'">
                          <sx:flatRecordType name='BEG'>
                                    <sx:fieldDelimiter value="|"/>
                                    <sx:delimitedField name="recordType"/>
                                    <sx:delimitedField name="id"/>
                                    <sx:delimitedField name="date"/>
                          </sx:flatRecordType>
                        </sx:when>
              </sx:flatRecordTypeChoice>
            </sx:flatFileBody>
  </sx:flatFile>

  <sx:recordMapping id="toXmlMapping">
            <records>
              <sx:groupBy fields="id">
                        <record>
                          <sx:fieldAttributeMap field="id" attribute="id"/>
                          <sx:fieldAttributeMap field="date" attribute="date"/>
                        </record>
              </sx:groupBy>
            </records>
  </sx:recordMapping>
</sx:resources>

This works well for small flat files but throws an out of memory error on a full dataset.
Is there a way to have it flush out the data after each record to avoid the problem?

Regards.