[Servingxml-help] Re: Multiple record types per row in flatfile
Brought to you by:
danielaparker
Menu
▾
▴
[Servingxml-help] Re: Multiple record types per row in flatfile
|
From: Daniel P. <dan...@sy...> - 2005-11-10 13:48:12
|
Wonne,
Have a look at the sx:removeEmptyElementFilter element in the element =
reference and in the examples "CONV" and "multivalued" where it is used. =
That will allow you to strip out the empty <info/> elements.
-- Daniel
----- Original Message -----=20
From: Keysers Wonne=20
To: Daniel Parker=20
Sent: Thursday, November 10, 2005 8:33 AM
Subject: RE: Multiple record types per row in flatfile
Hi Daniel,
Sorry to bother you again, but I encounter yet another issue in our =
file that I cannot solve 'instantly'.
Imagine that the persons.txt file contains 2 additional fields info1 =
and info2 and that they need to go into the PersonInfo section of the =
resulting file:
<PersonInfos>
<PersonInfo>
<PersonId/>
<Infos>
<Info><value/><sequence/></Info>
<Info><value/><sequence/></Info>
</Infos>
</PersonInfo>
<PersonInfos>
No problem so fa; the problem is that if info2 field is empty, the =
<Info/> element should not be generated and I don't seem to be able to =
do this with the grouping mechanism:
<sx:recordMapping name=3D"infosMapping">
<PersonInfos>
<sx:innerGroup>
<sx:startGroup test=3D"sx:current//DataType=3D'P'"/>
<sx:groupBy fields=3D"PersonId">
<PersonInfo>
<sx:fieldElementMap field=3D"PersonId" =
element=3D"PersonId"/>
<Infos>
<sx:groupChoice>
<sx:innerGroup>
<sx:startGroup test=3D"sx:current//info1!=3D''"/>
<Info><sx:fieldElementMap field=3D"info1" =
element=3D"Value"/></Info><Sequence>1</Sequence>
</sx:innerGroup>
<sx:innerGroup>
<sx:startGroup test=3D"sx:current//info2!=3D''"/>
<Info><sx:fieldElementMap field=3D"info2" =
element=3D"Value"/></Info><Sequence>2</Sequence>
</sx:innerGroup>
</Infos>
</sx:groupChoice>
</PersonInfo>
</sx:groupBy>
</sx:innerGroup>
</PersonInfos>
</sx:recordMapping>
Do you have a solution?
Thanks
Wonne
-------------------------------------------------------------------------=
-----
From: Daniel Parker [mailto:dan...@sy...]
Sent: Mon 7/11/2005 3:44
To: Keysers Wonne
Subject: Re: Multiple record types per row in flatfile
Hi Wonne,
Your record type definition is fine (there is a new approach =
documented, but the old approach is still supported, don't change it =
yet.)
This statement
<sx:onRecord recordType=3D"persons" />
only restricts the children of sx:onRecord, it has no affect on the =
ancestors (that's part of "no side effects.") So this won't help you.
There are two places where you can restrict the records. (1.) You =
can filter out records in the record reader, so that you'll never see =
them in the record mapping section. Or, (2.) you can restrict grouping =
to particular record types in the record mapping section.
To filter out records in the record reader (1), you can add a =
sx:restrictRecordFilter element to the children of sx:flatFileReader as =
shown below. Now the persons mapping won't see the DataType=3D'C' =
records at all, but the address mappings will.
<sx:recordMapping name=3D"personsAddressesMapping">
<Persons-Addresses>
<sx:recordContent>
<sx:flatFileReader>
<sx:flatFile ref=3D"personsData"/>
<sx:restrictRecordFilter>
<sx:restrictField field=3D"DataType" match=3D"P"/>
</sx:restrictRecordFilter>
</sx:flatFileReader>
<sx:recordMapping ref=3D"personsMapping"/>
</sx:recordContent>
<sx:recordContent>
<sx:flatFileReader>
<sx:flatFile ref=3D"personsData"/>
</sx:flatFileReader>
<sx:recordMapping ref=3D"addressesMapping"/>
</sx:recordContent>
</Persons-Addresses>
</sx:recordMapping>
The alternative (2) is to restrict the group to persons records. The =
simplest way would be to change the groupBy as follows
<sx:groupBy recordType=3D"persons" fields=3D"PersonId">
Unfortunately that's not currently supported (I'll add it to the next =
release.) =20
Instead you can enclose the sx:groupBy element in a sx:innerGroup =
element that only allows the persons through, like this
<sx:recordMapping name=3D"personsMapping">
<Persons>
<sx:innerGroup>
<sx:startGroup test=3D"boolean(sx:current/persons)=3D'true'"/>
<sx:groupBy fields=3D"PersonId">
<Person>
<sx:fieldElementMap field=3D"PersonId" =
element=3D"PersonId"/>
<sx:fieldElementMap field=3D"Name" element=3D"Name"/>
<sx:fieldElementMap field=3D"FirstName" =
element=3D"FirstName"/>
<sx:onRecord/>
</Person>
</sx:groupBy>
</sx:innerGroup>
</Persons>
</sx:recordMapping>
(You could also use=20
<sx:startGroup test=3D"sx:current//DataType=3D'P'"/>)
Regards,
Daniel Parker
----- Original Message -----=20
From: Keysers Wonne=20
To: Daniel Parker=20
Sent: Friday, November 04, 2005 3:00 AM
Subject: RE: Multiple record types per row in flatfile
Hi Daniel,
This feature works great when the flat file contains only persons, =
but now I have a file containing e.g. companies as well.
Example: The first field (DataType) in the flatfile indicates =
whether it is a person or a company. In case of a company, the FirstName =
is blank.
I was trying to implement this by changing the flatfile definition =
and persons record type from the example to something like this, but it =
doesn't seem to work.=20
<sx:flatFile name=3D"personsData">
<sx:flatFileHeader lineCount=3D"1"/>
<sx:flatFileBody>
<sx:flatRecordType>
<sx:positionalField name=3D"DataType" start=3D"1" =
width=3D"1"/>
</sx:flatRecordType>
<sx:choose>
<sx:when test=3D"DataType=3D'P'">
<sx:flatRecordType ref=3D"persons"/>
</sx:when>
<sx:otherwise><sx:flatRecordType =
ref=3D"companies"/></sx:otherwise>
</sx:choose>
</sx:flatFileBody>
</sx:flatFile>=20
<sx:recordMapping name=3D"personsMapping">
<Persons>
<sx:groupBy fields=3D"PersonId">
<Person>
<sx:fieldElementMap field=3D"PersonId" =
element=3D"PersonId"/>
<sx:fieldElementMap field=3D"Name" element=3D"Name"/>
<sx:fieldElementMap field=3D"FirstName" =
element=3D"FirstName"/>
<sx:onRecord recordType=3D"persons" />
</Person>
</sx:groupBy>
</Persons>
</sx:recordMapping>
Do you have a solution for this as well?
Thanks again
Wonne
-------------------------------------------------------------------------=
---
From: Daniel Parker [mailto:dan...@sy...]
Sent: Fri 28/10/2005 23:39
To: Keysers Wonne
Subject: Re: Multiple record types per row in flatfile
Hello Wonne,
I'll implement the proposed solution first, which I know how to do, =
and if you could test it that would be very much appreciated.
The alternative would mean having to support two groups of records =
below the root element, Persons and PersonAddresses, viz.
<Example>
<Persons> ...
</Persons>
<PersonAddresses> ...
</PersonAddresses> =20
</Example>
Currently, the tool only supports one, I don't know how to support =
two (with one record source) and still stream the records through, =
without caching the Persons and PersonAddresses output nodes.
Regards,
Daniel Parker
----- Original Message -----=20
From: Keysers Wonne=20
To: Daniel Parker=20
Sent: Friday, October 28, 2005 4:00 AM
Subject: RE: Multiple record types per row in flatfile
Hi Daniel,
Yes yes, this could indeed be a solution for our case, since 1 =
line is converted to multiple XML entities and grouping, etc. can be =
added to each recordMapping (e.g. all addresses grouped per person)
I'm only affraid that there will be a tremendous performance =
penalty when reading the flat file multiple times, because the real data =
files we are dealing with are quite big, if not huge, and they also =
contain even more record mappings.
Therefore, would it be a solution to allow multiple recordMappings =
on the same file? (This was in fact what I meant with 'multiple record =
types per row')=20
Anyway, if you think you have a release candidate, I'm certainly =
willing to test it with files from our data provider. The sooner the =
better :-)
Thanks, by the way, for your great support and framework!
Kind regards
Wonne
-------------------------------------------------------------------------=
-
From: Daniel Parker [mailto:dan...@sy...]
Sent: Fri 28/10/2005 6:12
To: Keysers Wonne
Subject: Re: Multiple record types per row in flatfile
Wonne,
Sorry, I revisited your original example and it looks like I =
misunderstood it. I do think the subject line is a little misleading! =
I read it as referring to repeating groups on a line.
Anyway, I believe the resources script below will meet your =
requirements, and will be supported in the next version.
<sx:resources xmlns:sx=3D"http://www.servingxml.com/core">
<sx:service name=3D"example">
<sx:serialize>
<sx:transform>
<sx:recordMapping ref=3D"exampleMapping"/>
</sx:transform>
</sx:serialize>
</sx:service>
<sx:flatFile name=3D"personsFlatFile">
<sx:flatFileHeader lineCount=3D"1"/>
<sx:flatFileBody>
<sx:flatRecordType ref=3D"persons"/>
</sx:flatFileBody>
</sx:flatFile>
<sx:flatRecordType name=3D"persons">
<sx:positionalField name=3D"PersonId" width=3D"9"/>
<sx:positionalField name=3D"Name" width=3D"11"/>
<sx:positionalField name=3D"FirstName" width=3D"16"/>
<sx:positionalField name=3D"Street" width=3D"13"/>
<sx:positionalField name=3D"PostCode" width=3D"9"/>
<sx:positionalField name=3D"CityTown" width=3D"9" />
</sx:flatRecordType>
<sx:recordMapping name=3D"exampleMapping">
<Example>
<sx:recordMapping>
<sx:flatFileReader><sx:flatFile =
ref=3D"personsFlatFile"/></sx:flatFileReader>
<sx:recordMapping ref=3D"personsMapping"/>
</sx:recordMapping>
<sx:recordMapping>
<sx:flatFileReader><sx:flatFile =
ref=3D"personsFlatFile"/></sx:flatFileReader>
<sx:recordMapping ref=3D"addressMapping"/>
</sx:recordMapping>
</Example>
</sx:recordMapping>
<sx:recordMapping name=3D"personsMapping">
<Persons>
<sx:onRecord>
<Person>
<sx:fieldElementMap field=3D"PersonId" =
element=3D"PersonId"/>
<sx:fieldElementMap field=3D"Name" element=3D"Name"/>
<sx:fieldElementMap field=3D"FirstName" =
element=3D"FirstName"/>
</Person>
</sx:onRecord>
</Persons>
</sx:recordMapping>
<sx:recordMapping name=3D"addressMapping">
<Addresses>
<sx:onRecord>
<Address>
<sx:fieldElementMap field=3D"PersonId" =
element=3D"PersonId"/>
<sx:fieldElementMap field=3D"Street" =
element=3D"Street"/>
<sx:fieldElementMap field=3D"PostCode" =
element=3D"PostCode"/>
<sx:fieldElementMap field=3D"CityTown" =
element=3D"CityTown"/>
</Address>
</sx:onRecord>
</Addresses>
</sx:recordMapping>
</sx:resources>
sx:recordMapping now supports the contract of both "content" as =
well as "mapping", and can be nested at any depth in the mapping tree. =
In its content, it can have a record reader(optional) and another record =
mapping element.
This is quite a nice use case, thanks. In your case, you read =
from the same file twice, but this also supports aggregating the content =
of different flat files, or a combination of flat file records and =
database records.
-- Daniel
----- Original Message -----=20
From: Daniel Parker=20
To: Keysers Wonne=20
Sent: Monday, October 24, 2005 11:27 PM
Subject: Re: Multiple record types per row in flatfile
Hello Wonne,
The feature is already implemented and has been tested with edi =
examples and examples having repeating fixed length segments within =
positional records; I don't anticipate any difficulties there. I should =
be able to have a new release out within two days of getting your sample =
data and expected output, assuming, of course, that no additional issues =
arise from your test case.
-- Daniel
----- Original Message -----=20
From: Keysers Wonne=20
To: dan...@us...=20
Sent: Monday, October 24, 2005 8:29 AM
Subject: RE: Multiple record types per row in flatfile
Hi Daniel,
Thank you for your quick response.
I was in the process of evaluating mechanisms to perform =
flatfile-to-xml conversions when I ran into the ServingXML project. The =
framework looks very flexible and enhanced and could bring a lot of =
added value to our project, but we are, however, a bit blocked with this =
missing feature.
Since I am currently under preassure to decide upon the =
technologies to use, could you please share me your estimate of when you =
think to be able to release the new feature?
If you think it is feasible in a short amount of time, I can =
assist you in creating example data and, if necessary, do some coding. =
Otherwise I'm unfortunately forced to choose other options. Please don't =
interprete this as a threat, I'm not happy with it either...
Can you please give me your thoughts?
Thanks in advance
Wonne
----------------------------------------------------------------------
From: Daniel Parker
Subject: Multiple record types per row in flatfile
Hello Keysers,
As it happens, the next version, due shortly, will support =
repeating groups of record segments on one row, which should answer your =
needs. I'll include your example below in the Examples. Would you be =
able to provide a sample flat file with some representative groups of =
addresses? =20
Thanks,
Daniel Parker
----------------------------------------------------------------------
From: Keysers Wonne
Sent: Fri 21/10/2005 9:26
To: ser...@li...
Subject: Multiple record types per row in flatfile
Hi,
I need to parse and transform a flatfile (fixed length) file =
into XML.
The flatfile contains multiple kinds of records on one single =
row and they all go into separate sections in the XML.
Here is a small example of a person with address data on the =
same line. An important side effect is that a person may have multiple =
lines in the flatfile indicating multiple addresses. (e.g. professional =
/ private / ...)
FlatFile (Positional):
PersonId PersonName PersonFirstName Street Postcode CityTown
Resulting XML file should look like:
<Example>
<Persons>
<Person>
<PersonId/>
<Name/>
<FirstName/>
</Person>
</Persons>
<PersonAddresses>
<PersonAddress>
<PersonId/>
<Addresses>
<Addres>
<Street/>
<PostCode/>
<CityTown/>
<Addres>
</Addresses>
</PersonAddress>
</PersonAddresses> =20
</Example>
The problem could be solved when multiple recordContent =
elements could be specified for a row, but after reading the online =
documentation, I don't think this is currently possible?
What solution do you propose?
Thanks in advance!
/Wonne |
