Thread: [Servingxml-help] Flat file structure validation
Brought to you by:
danielaparker
|
From: Ravikumar T. <ra...@gm...> - 2007-03-09 04:16:58
|
Hi I am trying to use servingxml to validate and transform flat files to xml. I am wondering if this toolkit can help validate the flat file which looks as follows: Every row is a record with position #1 as record type. Position #2-5 as filenumber and the rest as misc fields. I want to validate that for every filenumber, there exists atleast 1 E record followed by 0 or more S records. E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi |
|
From: Daniel P. <dan...@sy...> - 2007-03-13 01:29:26
|
Hello Ravi, I think you'd have to transform the flat file to XML and express the = rules in an XML Schema, using the msv:schemaValidator (previously = msv:msvFilter) to validate the XML. Alternatively, you could write a = custom record filter in Java, and buffer enough records until you could = determine whether they were valid, and discard them or throw an = exception if you find an error. But other than that, the tool doesn't = provide anything special for validating groups of records, as opposed to = individual records. Best regards, Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: ser...@li...=20 Sent: Friday, March 09, 2007 12:16 AM Subject: [Servingxml-help] Flat file structure validation Hi I am trying to use servingxml to validate and transform flat files to = xml. I am wondering if this toolkit can help validate the flat file = which looks as follows: Every row is a record with position #1 as record type. Position #2-5 = as filenumber and the rest as misc fields. I want to validate that for = every filenumber, there exists atleast 1 E record followed by 0 or more = S records.=20 E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi -------------------------------------------------------------------------= ----- = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to = share your opinions on IT & business topics through brief surveys-and earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV -------------------------------------------------------------------------= ----- _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help |
|
From: Daniel P. <dan...@sy...> - 2007-03-23 02:35:22
|
Yes, in record mapping sections you can define groupings based on the =
previous, current, and next records, check out any of the examples using =
sx:innerGroup. The case below is no problem, I may use it in as example =
in the element reference. =20
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: Daniel Parker=20
Sent: Thursday, March 22, 2007 4:06 PM
Subject: Re: [Servingxml-help] Flat file structure validation
Thanks Daniel. This is helpful.=20
Another question on the same file structure.
It looks like serving xml processes one record (using onRecord) at a =
time. Can I have control over 2 record at a time while converting flat =
file to xml. I want to convert the below flat file as follows. =
Essentially I want to put one field from E record and one field from S =
record into <group1>. So while processing E record, I need information =
from the next record also.=20
E1001FIELD0
S1001FIELD1FIELD2
E1002FIELDA
S1002FIELDBFIELDC
TO
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
<group2>
<c>FIELD1</c>
</group2>
</record>
On 3/12/07, Daniel Parker <dan...@sy...> wrote:
Hello Ravi,
I think you'd have to transform the flat file to XML and express the =
rules in an XML Schema, using the msv:schemaValidator (previously =
msv:msvFilter) to validate the XML. Alternatively, you could write a =
custom record filter in Java, and buffer enough records until you could =
determine whether they were valid, and discard them or throw an =
exception if you find an error. But other than that, the tool doesn't =
provide anything special for validating groups of records, as opposed to =
individual records.
Best regards,
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: ser...@li...=20
Sent: Friday, March 09, 2007 12:16 AM
Subject: [Servingxml-help] Flat file structure validation
Hi
I am trying to use servingxml to validate and transform flat files =
to xml. I am wondering if this toolkit can help validate the flat file =
which looks as follows:
Every row is a record with position #1 as record type. Position =
#2-5 as filenumber and the rest as misc fields. I want to validate that =
for every filenumber, there exists atleast 1 E record followed by 0 or =
more S records.=20
E1001FIELD0
S1001FIELD1FIELD2
S1001FIELD3FIELD4
E1002FIELDA
S1002FIELDBFIELDC
Thanks
Ravi
-------------------------------------------------------------------------=
-
=
-------------------------------------------------------------------------=
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to =
share your
opinions on IT & business topics through brief surveys-and earn =
cash
=
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV=20
-------------------------------------------------------------------------=
-
_______________________________________________
Servingxml-help mailing list
Ser...@li...
https://lists.sourceforge.net/lists/listinfo/servingxml-help
|
|
From: Daniel P. <dan...@sy...> - 2007-03-28 02:42:58
|
This isn't currently supported, although it could easily be added.=20
The problem is that you have two groups, and the second one immediately =
follows the first one - its first record is the first one's last record. =
Currently, the two groups must belong to a sx:recordChoice, so any =
given record can belong to only one group at the same level. However, =
since the first group is finished before the second group begins, I =
believe it's possible to relax the constraints to allow this. =20
Because grouping is streaming, the records aren't buffered, care has to =
be taken not to start a second group before the first one ends, or the =
output tags will be scrambled. That was the reason for restricting one =
record to only one group at the same level, to prevent surprises, but it =
looks like this restriction is too strong. I'll have a look at it.
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: Daniel Parker=20
Sent: Sunday, March 25, 2007 10:57 PM
Subject: Re: [Servingxml-help] Flat file structure validation
Daniel,
sorry for trouble. I am unable to achieve what I wanted to do. I was =
trying this through the weekend. I am able to achieve only a part of =
expected output. I am sending my resource script, input file, expected =
output and actual output. I hope you will help me fix the resource =
script to get the expected output. I am sending the resource script as =
attachment.=20
Input - ytds.txt
E1002FIELD0
S1002FIELD1FIELD2
ExpectedOutput:
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
<group2>
<c>FIELD1</c>
</group2>
</record>
ActualOutput:
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
</record>
Regards
Ravi
On 3/22/07, Daniel Parker <dan...@sy... > wrote:=20
Yes, in record mapping sections you can define groupings based on =
the previous, current, and next records, check out any of the examples =
using sx:innerGroup. The case below is no problem, I may use it in as =
example in the element reference. =20
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: Daniel Parker=20
Sent: Thursday, March 22, 2007 4:06 PM
Subject: Re: [Servingxml-help] Flat file structure validation
Thanks Daniel. This is helpful.=20
Another question on the same file structure.
It looks like serving xml processes one record (using onRecord) at =
a time. Can I have control over 2 record at a time while converting flat =
file to xml. I want to convert the below flat file as follows. =
Essentially I want to put one field from E record and one field from S =
record into <group1>. So while processing E record, I need information =
from the next record also.=20
E1001FIELD0
S1001FIELD1FIELD2
E1002FIELDA
S1002FIELDBFIELDC
TO
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
<group2>
<c>FIELD1</c>
</group2>
</record>
On 3/12/07, Daniel Parker <dan...@sy...> wrote:=20
Hello Ravi,
I think you'd have to transform the flat file to XML and express =
the rules in an XML Schema, using the msv:schemaValidator (previously =
msv:msvFilter) to validate the XML. Alternatively, you could write a =
custom record filter in Java, and buffer enough records until you could =
determine whether they were valid, and discard them or throw an =
exception if you find an error. But other than that, the tool doesn't =
provide anything special for validating groups of records, as opposed to =
individual records.
Best regards,
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: ser...@li...=20
Sent: Friday, March 09, 2007 12:16 AM
Subject: [Servingxml-help] Flat file structure validation
Hi
I am trying to use servingxml to validate and transform flat =
files to xml. I am wondering if this toolkit can help validate the flat =
file which looks as follows:
Every row is a record with position #1 as record type. =
Position #2-5 as filenumber and the rest as misc fields. I want to =
validate that for every filenumber, there exists atleast 1 E record =
followed by 0 or more S records.=20
E1001FIELD0
S1001FIELD1FIELD2
S1001FIELD3FIELD4
E1002FIELDA
S1002FIELDBFIELDC
Thanks
Ravi
----------------------------------------------------------------------
=
-------------------------------------------------------------------------=
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance =
to share your
opinions on IT & business topics through brief surveys-and =
earn cash
=
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV=20
----------------------------------------------------------------------
_______________________________________________
Servingxml-help mailing list
Ser...@li...
https://lists.sourceforge.net/lists/listinfo/servingxml-help
=
-------------------------------------------------------------------------=
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to =
share your=20
opinions on IT & business topics through brief surveys-and earn cash
=
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
_______________________________________________
Servingxml-help mailing list
Ser...@li...
https://lists.sourceforge.net/lists/listinfo/servingxml-help=20
|
|
From: Daniel P. <dan...@sy...> - 2007-04-30 03:55:57
|
Ravi,
This is now supported in version 0.6.6. Refer to the element reference =
entry for sx:innerGroup, to the last example.
-- Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: Daniel Parker=20
Sent: Sunday, March 25, 2007 10:57 PM
Subject: Re: [Servingxml-help] Flat file structure validation
Daniel,
sorry for trouble. I am unable to achieve what I wanted to do. I was =
trying this through the weekend. I am able to achieve only a part of =
expected output. I am sending my resource script, input file, expected =
output and actual output. I hope you will help me fix the resource =
script to get the expected output. I am sending the resource script as =
attachment.=20
Input - ytds.txt
E1002FIELD0
S1002FIELD1FIELD2
ExpectedOutput:
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
<group2>
<c>FIELD1</c>
</group2>
</record>
ActualOutput:
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
</record>
Regards
Ravi
On 3/22/07, Daniel Parker <dan...@sy... > wrote:
Yes, in record mapping sections you can define groupings based on =
the previous, current, and next records, check out any of the examples =
using sx:innerGroup. The case below is no problem, I may use it in as =
example in the element reference. =20
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: Daniel Parker=20
Sent: Thursday, March 22, 2007 4:06 PM
Subject: Re: [Servingxml-help] Flat file structure validation
Thanks Daniel. This is helpful.=20
Another question on the same file structure.
It looks like serving xml processes one record (using onRecord) at =
a time. Can I have control over 2 record at a time while converting flat =
file to xml. I want to convert the below flat file as follows. =
Essentially I want to put one field from E record and one field from S =
record into <group1>. So while processing E record, I need information =
from the next record also.=20
E1001FIELD0
S1001FIELD1FIELD2
E1002FIELDA
S1002FIELDBFIELDC
TO
<record fileno=3D1001>
<group1>
<a>FIELD0</a>=20
<b>FIELD2</b>
</group1>
<group2>
<c>FIELD1</c>
</group2>
</record>
On 3/12/07, Daniel Parker <dan...@sy...> wrote:=20
Hello Ravi,
I think you'd have to transform the flat file to XML and express =
the rules in an XML Schema, using the msv:schemaValidator (previously =
msv:msvFilter) to validate the XML. Alternatively, you could write a =
custom record filter in Java, and buffer enough records until you could =
determine whether they were valid, and discard them or throw an =
exception if you find an error. But other than that, the tool doesn't =
provide anything special for validating groups of records, as opposed to =
individual records.
Best regards,
Daniel
----- Original Message -----=20
From: Ravikumar Tadysetty=20
To: ser...@li...=20
Sent: Friday, March 09, 2007 12:16 AM
Subject: [Servingxml-help] Flat file structure validation
Hi
I am trying to use servingxml to validate and transform flat =
files to xml. I am wondering if this toolkit can help validate the flat =
file which looks as follows:
Every row is a record with position #1 as record type. =
Position #2-5 as filenumber and the rest as misc fields. I want to =
validate that for every filenumber, there exists atleast 1 E record =
followed by 0 or more S records.=20
E1001FIELD0
S1001FIELD1FIELD2
S1001FIELD3FIELD4
E1002FIELDA
S1002FIELDBFIELDC
Thanks
Ravi
----------------------------------------------------------------------
=
-------------------------------------------------------------------------=
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance =
to share your
opinions on IT & business topics through brief surveys-and =
earn cash
=
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV=20
----------------------------------------------------------------------
_______________________________________________
Servingxml-help mailing list
Ser...@li...
https://lists.sourceforge.net/lists/listinfo/servingxml-help
=
-------------------------------------------------------------------------=
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to =
share your=20
opinions on IT & business topics through brief surveys-and earn cash
=
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV
_______________________________________________
Servingxml-help mailing list
Ser...@li...
https://lists.sourceforge.net/lists/listinfo/servingxml-help=20
|
|
From: Ravikumar T. <ra...@gm...> - 2007-05-11 03:56:24
|
yup. That is exactly what we are doing today. On 5/10/07, Daniel Parker <ser...@li...> wrote: > > Sorry for the delayed response. > > Buffering isn't as simple as that. It has to be optional, since some > users are using this tool with very large datasets. Also, note that record > filters can remove some records from a stream, so what was once the previous > record is no longer. > > For now, I would suggest using a custom record filter, do your own > buffering, and modify the current record by adding the fields you need from > previous records. > > Daniel > > ----- Original Message ----- > *From:* Ravikumar Tadysetty <ra...@gm...> > *To:* Daniel Parker <dan...@sy...> > *Sent:* Sunday, May 06, 2007 9:03 PM > *Subject:* Re: [Servingxml-help] Flat file structure validation > > Daniel > > For situations like this one (or more complicated than this), can we do > something like: > > group all records by file number. > Hold all those records in a memory buffer. > Provide a facility to access that memory buffer in the record mapping > section. > eg: > > <group1> > <a> ERecord->[0] </a> > <b> SRecord->[1] </b> > <group1> > <group2> > <c>SRecord->[0]</c> > <d>ERecord->[1]</d> > </group2> > > This way we can access data of any record in that group from any element > of the xml. One of the main constraints today is that servingxml mapping > processes one record at a time. Once that record is processed, it cannot be > accessed further. We donot have the ability to look-ahead or look-behind > while processing a record in a group. > > > On 5/5/07, Ravikumar Tadysetty <ra...@gm...> wrote: > > > > Daniel, Thank you. Really waiting for this. However this was only a part > > of the puzzle. I wanted more flexibility. Please see the below example. I > > added one more field and I cannot achieve this now. Can you suggest > > something for this type of situation. Essentially, while printing <group2> I > > want the data from the previous record. > > > > From the previous example, I added 2 new fields FIELDX, FIELDY. I want > > to put FIELDX from ERecord and FIELD1 from SRecord in <group2>. > > > > FROM > > > > E1001FIELD0FIELDX > > S1001FIELD1FIELD2 > > > > E1002FIELDAFIELDY > > > > S1002FIELDBFIELDC > > > > TO > > > > <ytds> > > <record fileno="1001"> > > <group1> > > <a>FIELD0</a> > > <b>FIELD2</b> > > </group1> > > > > > > <group2> > > <c>FIELD1</c> > > <d>FIELDX</d> > > </group2> > > </record> > > <record fileno="1002"> > > <group1> > > <a>FIELDA</a> > > > > > > <b>FIELDC</b> > > </group1> > > <group2> > > <c>FIELDB</c> > > <d>FIELDY</d> > > </group2> > > </record> > > </ytds> > > > > > > > > On 4/29/07, Daniel Parker <dan...@sy... > wrote: > > > > > > Ravi, > > > > > > This is now supported in version 0.6.6. Refer to the element > > > reference entry for sx:innerGroup, to the last example. > > > > > > -- Daniel > > > > > > ----- Original Message ----- > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > *To:* Daniel Parker <dan...@sy...> > > > *Sent:* Sunday, March 25, 2007 10:57 PM > > > *Subject:* Re: [Servingxml-help] Flat file structure validation > > > > > > Daniel, > > > > > > sorry for trouble. I am unable to achieve what I wanted to do. I was > > > trying this through the weekend. I am able to achieve only a part of > > > expected output. I am sending my resource script, input file, expected > > > output and actual output. I hope you will help me fix the resource script to > > > get the expected output. I am sending the resource script as attachment. > > > > > > Input - ytds.txt > > > > > > E1002FIELD0 > > > S1002FIELD1FIELD2 > > > > > > ExpectedOutput: > > > > > > <record fileno=1001> > > > <group1> > > > <a>FIELD0</a> > > > <b>FIELD2</b> > > > </group1> > > > <group2> > > > <c>FIELD1</c> > > > </group2> > > > </record> > > > > > > ActualOutput: > > > > > > <record fileno=1001> > > > <group1> > > > <a>FIELD0</a> > > > <b>FIELD2</b> > > > </group1> > > > </record> > > > > > > Regards > > > Ravi > > > > > > On 3/22/07, Daniel Parker <dan...@sy... > wrote: > > > > > > > > Yes, in record mapping sections you can define groupings based on > > > > the previous, current, and next records, check out any of the examples using > > > > sx:innerGroup. The case below is no problem, I may use it in as example in > > > > the element reference. > > > > > > > > Daniel > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > > *To:* Daniel Parker <dan...@sy...> > > > > *Sent:* Thursday, March 22, 2007 4:06 PM > > > > *Subject:* Re: [Servingxml-help] Flat file structure validation > > > > > > > > Thanks Daniel. This is helpful. > > > > Another question on the same file structure. > > > > > > > > It looks like serving xml processes one record (using onRecord) at a > > > > time. Can I have control over 2 record at a time while converting flat file > > > > to xml. I want to convert the below flat file as follows. Essentially I > > > > want to put one field from E record and one field from S record into > > > > <group1>. So while processing E record, I need information from the next > > > > record also. > > > > > > > > E1001FIELD0 > > > > S1001FIELD1FIELD2 > > > > E1002FIELDA > > > > S1002FIELDBFIELDC > > > > > > > > TO > > > > > > > > <record fileno=1001> > > > > <group1> > > > > <a>FIELD0</a> > > > > <b>FIELD2</b> > > > > </group1> > > > > <group2> > > > > <c>FIELD1</c> > > > > </group2> > > > > </record> > > > > > > > > > > > > > > > > > > > > On 3/12/07, Daniel Parker <dan...@sy...> wrote: > > > > > > > > > > Hello Ravi, > > > > > > > > > > I think you'd have to transform the flat file to XML and express > > > > > the rules in an XML Schema, using the msv:schemaValidator (previously > > > > > msv:msvFilter) to validate the XML. Alternatively, you could write a custom > > > > > record filter in Java, and buffer enough records until you could determine > > > > > whether they were valid, and discard them or throw an exception if you find > > > > > an error. But other than that, the tool doesn't provide anything special > > > > > for validating groups of records, as opposed to individual records. > > > > > > > > > > Best regards, > > > > > Daniel > > > > > > > > > > ----- Original Message ----- > > > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > > > *To:* ser...@li... > > > > > *Sent:* Friday, March 09, 2007 12:16 AM > > > > > *Subject:* [Servingxml-help] Flat file structure validation > > > > > > > > > > Hi > > > > > > > > > > I am trying to use servingxml to validate and transform flat files > > > > > to xml. I am wondering if this toolkit can help validate the flat file which > > > > > looks as follows: > > > > > > > > > > Every row is a record with position #1 as record type. Position > > > > > #2-5 as filenumber and the rest as misc fields. I want to validate that for > > > > > every filenumber, there exists atleast 1 E record followed by 0 or more S > > > > > records. > > > > > > > > > > E1001FIELD0 > > > > > S1001FIELD1FIELD2 > > > > > S1001FIELD3FIELD4 > > > > > E1002FIELDA > > > > > S1002FIELDBFIELDC > > > > > > > > > > Thanks > > > > > Ravi > > > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > > > > share your > > > > > opinions on IT & business topics through brief surveys-and earn > > > > > cash > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > > > > > > > > ------------------------------ > > > > > > > > > > _______________________________________________ > > > > > Servingxml-help mailing list > > > > > Ser...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/servingxml-help > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > > > share your > > > > opinions on IT & business topics through brief surveys-and earn cash > > > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > _______________________________________________ > > > > Servingxml-help mailing list > > > > Ser...@li... > > > > https://lists.sourceforge.net/lists/listinfo/servingxml-help > > > > > > > > > > > > > > |