Thread: [Servingxml-help] Flat file structure validation
Brought to you by:
danielaparker
From: Ravikumar T. <ra...@gm...> - 2007-03-09 04:16:58
|
Hi I am trying to use servingxml to validate and transform flat files to xml. I am wondering if this toolkit can help validate the flat file which looks as follows: Every row is a record with position #1 as record type. Position #2-5 as filenumber and the rest as misc fields. I want to validate that for every filenumber, there exists atleast 1 E record followed by 0 or more S records. E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi |
From: Daniel P. <dan...@sy...> - 2007-03-13 01:29:26
|
Hello Ravi, I think you'd have to transform the flat file to XML and express the = rules in an XML Schema, using the msv:schemaValidator (previously = msv:msvFilter) to validate the XML. Alternatively, you could write a = custom record filter in Java, and buffer enough records until you could = determine whether they were valid, and discard them or throw an = exception if you find an error. But other than that, the tool doesn't = provide anything special for validating groups of records, as opposed to = individual records. Best regards, Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: ser...@li...=20 Sent: Friday, March 09, 2007 12:16 AM Subject: [Servingxml-help] Flat file structure validation Hi I am trying to use servingxml to validate and transform flat files to = xml. I am wondering if this toolkit can help validate the flat file = which looks as follows: Every row is a record with position #1 as record type. Position #2-5 = as filenumber and the rest as misc fields. I want to validate that for = every filenumber, there exists atleast 1 E record followed by 0 or more = S records.=20 E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi -------------------------------------------------------------------------= ----- = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to = share your opinions on IT & business topics through brief surveys-and earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV -------------------------------------------------------------------------= ----- _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help |
From: Daniel P. <dan...@sy...> - 2007-03-23 02:35:22
|
Yes, in record mapping sections you can define groupings based on the = previous, current, and next records, check out any of the examples using = sx:innerGroup. The case below is no problem, I may use it in as example = in the element reference. =20 Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: Daniel Parker=20 Sent: Thursday, March 22, 2007 4:06 PM Subject: Re: [Servingxml-help] Flat file structure validation Thanks Daniel. This is helpful.=20 Another question on the same file structure. It looks like serving xml processes one record (using onRecord) at a = time. Can I have control over 2 record at a time while converting flat = file to xml. I want to convert the below flat file as follows. = Essentially I want to put one field from E record and one field from S = record into <group1>. So while processing E record, I need information = from the next record also.=20 E1001FIELD0 S1001FIELD1FIELD2 E1002FIELDA S1002FIELDBFIELDC TO <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> <group2> <c>FIELD1</c> </group2> </record> On 3/12/07, Daniel Parker <dan...@sy...> wrote: Hello Ravi, I think you'd have to transform the flat file to XML and express the = rules in an XML Schema, using the msv:schemaValidator (previously = msv:msvFilter) to validate the XML. Alternatively, you could write a = custom record filter in Java, and buffer enough records until you could = determine whether they were valid, and discard them or throw an = exception if you find an error. But other than that, the tool doesn't = provide anything special for validating groups of records, as opposed to = individual records. Best regards, Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: ser...@li...=20 Sent: Friday, March 09, 2007 12:16 AM Subject: [Servingxml-help] Flat file structure validation Hi I am trying to use servingxml to validate and transform flat files = to xml. I am wondering if this toolkit can help validate the flat file = which looks as follows: Every row is a record with position #1 as record type. Position = #2-5 as filenumber and the rest as misc fields. I want to validate that = for every filenumber, there exists atleast 1 E record followed by 0 or = more S records.=20 E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi -------------------------------------------------------------------------= - = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to = share your opinions on IT & business topics through brief surveys-and earn = cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV=20 -------------------------------------------------------------------------= - _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help |
From: Daniel P. <dan...@sy...> - 2007-03-28 02:42:58
|
This isn't currently supported, although it could easily be added.=20 The problem is that you have two groups, and the second one immediately = follows the first one - its first record is the first one's last record. = Currently, the two groups must belong to a sx:recordChoice, so any = given record can belong to only one group at the same level. However, = since the first group is finished before the second group begins, I = believe it's possible to relax the constraints to allow this. =20 Because grouping is streaming, the records aren't buffered, care has to = be taken not to start a second group before the first one ends, or the = output tags will be scrambled. That was the reason for restricting one = record to only one group at the same level, to prevent surprises, but it = looks like this restriction is too strong. I'll have a look at it. Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: Daniel Parker=20 Sent: Sunday, March 25, 2007 10:57 PM Subject: Re: [Servingxml-help] Flat file structure validation Daniel, sorry for trouble. I am unable to achieve what I wanted to do. I was = trying this through the weekend. I am able to achieve only a part of = expected output. I am sending my resource script, input file, expected = output and actual output. I hope you will help me fix the resource = script to get the expected output. I am sending the resource script as = attachment.=20 Input - ytds.txt E1002FIELD0 S1002FIELD1FIELD2 ExpectedOutput: <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> <group2> <c>FIELD1</c> </group2> </record> ActualOutput: <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> </record> Regards Ravi On 3/22/07, Daniel Parker <dan...@sy... > wrote:=20 Yes, in record mapping sections you can define groupings based on = the previous, current, and next records, check out any of the examples = using sx:innerGroup. The case below is no problem, I may use it in as = example in the element reference. =20 Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: Daniel Parker=20 Sent: Thursday, March 22, 2007 4:06 PM Subject: Re: [Servingxml-help] Flat file structure validation Thanks Daniel. This is helpful.=20 Another question on the same file structure. It looks like serving xml processes one record (using onRecord) at = a time. Can I have control over 2 record at a time while converting flat = file to xml. I want to convert the below flat file as follows. = Essentially I want to put one field from E record and one field from S = record into <group1>. So while processing E record, I need information = from the next record also.=20 E1001FIELD0 S1001FIELD1FIELD2 E1002FIELDA S1002FIELDBFIELDC TO <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> <group2> <c>FIELD1</c> </group2> </record> On 3/12/07, Daniel Parker <dan...@sy...> wrote:=20 Hello Ravi, I think you'd have to transform the flat file to XML and express = the rules in an XML Schema, using the msv:schemaValidator (previously = msv:msvFilter) to validate the XML. Alternatively, you could write a = custom record filter in Java, and buffer enough records until you could = determine whether they were valid, and discard them or throw an = exception if you find an error. But other than that, the tool doesn't = provide anything special for validating groups of records, as opposed to = individual records. Best regards, Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: ser...@li...=20 Sent: Friday, March 09, 2007 12:16 AM Subject: [Servingxml-help] Flat file structure validation Hi I am trying to use servingxml to validate and transform flat = files to xml. I am wondering if this toolkit can help validate the flat = file which looks as follows: Every row is a record with position #1 as record type. = Position #2-5 as filenumber and the rest as misc fields. I want to = validate that for every filenumber, there exists atleast 1 E record = followed by 0 or more S records.=20 E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi ---------------------------------------------------------------------- = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance = to share your opinions on IT & business topics through brief surveys-and = earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV=20 ---------------------------------------------------------------------- _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to = share your=20 opinions on IT & business topics through brief surveys-and earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help=20 |
From: Daniel P. <dan...@sy...> - 2007-04-30 03:55:57
|
Ravi, This is now supported in version 0.6.6. Refer to the element reference = entry for sx:innerGroup, to the last example. -- Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: Daniel Parker=20 Sent: Sunday, March 25, 2007 10:57 PM Subject: Re: [Servingxml-help] Flat file structure validation Daniel, sorry for trouble. I am unable to achieve what I wanted to do. I was = trying this through the weekend. I am able to achieve only a part of = expected output. I am sending my resource script, input file, expected = output and actual output. I hope you will help me fix the resource = script to get the expected output. I am sending the resource script as = attachment.=20 Input - ytds.txt E1002FIELD0 S1002FIELD1FIELD2 ExpectedOutput: <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> <group2> <c>FIELD1</c> </group2> </record> ActualOutput: <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> </record> Regards Ravi On 3/22/07, Daniel Parker <dan...@sy... > wrote: Yes, in record mapping sections you can define groupings based on = the previous, current, and next records, check out any of the examples = using sx:innerGroup. The case below is no problem, I may use it in as = example in the element reference. =20 Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: Daniel Parker=20 Sent: Thursday, March 22, 2007 4:06 PM Subject: Re: [Servingxml-help] Flat file structure validation Thanks Daniel. This is helpful.=20 Another question on the same file structure. It looks like serving xml processes one record (using onRecord) at = a time. Can I have control over 2 record at a time while converting flat = file to xml. I want to convert the below flat file as follows. = Essentially I want to put one field from E record and one field from S = record into <group1>. So while processing E record, I need information = from the next record also.=20 E1001FIELD0 S1001FIELD1FIELD2 E1002FIELDA S1002FIELDBFIELDC TO <record fileno=3D1001> <group1> <a>FIELD0</a>=20 <b>FIELD2</b> </group1> <group2> <c>FIELD1</c> </group2> </record> On 3/12/07, Daniel Parker <dan...@sy...> wrote:=20 Hello Ravi, I think you'd have to transform the flat file to XML and express = the rules in an XML Schema, using the msv:schemaValidator (previously = msv:msvFilter) to validate the XML. Alternatively, you could write a = custom record filter in Java, and buffer enough records until you could = determine whether they were valid, and discard them or throw an = exception if you find an error. But other than that, the tool doesn't = provide anything special for validating groups of records, as opposed to = individual records. Best regards, Daniel ----- Original Message -----=20 From: Ravikumar Tadysetty=20 To: ser...@li...=20 Sent: Friday, March 09, 2007 12:16 AM Subject: [Servingxml-help] Flat file structure validation Hi I am trying to use servingxml to validate and transform flat = files to xml. I am wondering if this toolkit can help validate the flat = file which looks as follows: Every row is a record with position #1 as record type. = Position #2-5 as filenumber and the rest as misc fields. I want to = validate that for every filenumber, there exists atleast 1 E record = followed by 0 or more S records.=20 E1001FIELD0 S1001FIELD1FIELD2 S1001FIELD3FIELD4 E1002FIELDA S1002FIELDBFIELDC Thanks Ravi ---------------------------------------------------------------------- = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance = to share your opinions on IT & business topics through brief surveys-and = earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV=20 ---------------------------------------------------------------------- _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help = -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to = share your=20 opinions on IT & business topics through brief surveys-and earn cash = http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV _______________________________________________ Servingxml-help mailing list Ser...@li... https://lists.sourceforge.net/lists/listinfo/servingxml-help=20 |
From: Ravikumar T. <ra...@gm...> - 2007-05-11 03:56:24
|
yup. That is exactly what we are doing today. On 5/10/07, Daniel Parker <ser...@li...> wrote: > > Sorry for the delayed response. > > Buffering isn't as simple as that. It has to be optional, since some > users are using this tool with very large datasets. Also, note that record > filters can remove some records from a stream, so what was once the previous > record is no longer. > > For now, I would suggest using a custom record filter, do your own > buffering, and modify the current record by adding the fields you need from > previous records. > > Daniel > > ----- Original Message ----- > *From:* Ravikumar Tadysetty <ra...@gm...> > *To:* Daniel Parker <dan...@sy...> > *Sent:* Sunday, May 06, 2007 9:03 PM > *Subject:* Re: [Servingxml-help] Flat file structure validation > > Daniel > > For situations like this one (or more complicated than this), can we do > something like: > > group all records by file number. > Hold all those records in a memory buffer. > Provide a facility to access that memory buffer in the record mapping > section. > eg: > > <group1> > <a> ERecord->[0] </a> > <b> SRecord->[1] </b> > <group1> > <group2> > <c>SRecord->[0]</c> > <d>ERecord->[1]</d> > </group2> > > This way we can access data of any record in that group from any element > of the xml. One of the main constraints today is that servingxml mapping > processes one record at a time. Once that record is processed, it cannot be > accessed further. We donot have the ability to look-ahead or look-behind > while processing a record in a group. > > > On 5/5/07, Ravikumar Tadysetty <ra...@gm...> wrote: > > > > Daniel, Thank you. Really waiting for this. However this was only a part > > of the puzzle. I wanted more flexibility. Please see the below example. I > > added one more field and I cannot achieve this now. Can you suggest > > something for this type of situation. Essentially, while printing <group2> I > > want the data from the previous record. > > > > From the previous example, I added 2 new fields FIELDX, FIELDY. I want > > to put FIELDX from ERecord and FIELD1 from SRecord in <group2>. > > > > FROM > > > > E1001FIELD0FIELDX > > S1001FIELD1FIELD2 > > > > E1002FIELDAFIELDY > > > > S1002FIELDBFIELDC > > > > TO > > > > <ytds> > > <record fileno="1001"> > > <group1> > > <a>FIELD0</a> > > <b>FIELD2</b> > > </group1> > > > > > > <group2> > > <c>FIELD1</c> > > <d>FIELDX</d> > > </group2> > > </record> > > <record fileno="1002"> > > <group1> > > <a>FIELDA</a> > > > > > > <b>FIELDC</b> > > </group1> > > <group2> > > <c>FIELDB</c> > > <d>FIELDY</d> > > </group2> > > </record> > > </ytds> > > > > > > > > On 4/29/07, Daniel Parker <dan...@sy... > wrote: > > > > > > Ravi, > > > > > > This is now supported in version 0.6.6. Refer to the element > > > reference entry for sx:innerGroup, to the last example. > > > > > > -- Daniel > > > > > > ----- Original Message ----- > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > *To:* Daniel Parker <dan...@sy...> > > > *Sent:* Sunday, March 25, 2007 10:57 PM > > > *Subject:* Re: [Servingxml-help] Flat file structure validation > > > > > > Daniel, > > > > > > sorry for trouble. I am unable to achieve what I wanted to do. I was > > > trying this through the weekend. I am able to achieve only a part of > > > expected output. I am sending my resource script, input file, expected > > > output and actual output. I hope you will help me fix the resource script to > > > get the expected output. I am sending the resource script as attachment. > > > > > > Input - ytds.txt > > > > > > E1002FIELD0 > > > S1002FIELD1FIELD2 > > > > > > ExpectedOutput: > > > > > > <record fileno=1001> > > > <group1> > > > <a>FIELD0</a> > > > <b>FIELD2</b> > > > </group1> > > > <group2> > > > <c>FIELD1</c> > > > </group2> > > > </record> > > > > > > ActualOutput: > > > > > > <record fileno=1001> > > > <group1> > > > <a>FIELD0</a> > > > <b>FIELD2</b> > > > </group1> > > > </record> > > > > > > Regards > > > Ravi > > > > > > On 3/22/07, Daniel Parker <dan...@sy... > wrote: > > > > > > > > Yes, in record mapping sections you can define groupings based on > > > > the previous, current, and next records, check out any of the examples using > > > > sx:innerGroup. The case below is no problem, I may use it in as example in > > > > the element reference. > > > > > > > > Daniel > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > > *To:* Daniel Parker <dan...@sy...> > > > > *Sent:* Thursday, March 22, 2007 4:06 PM > > > > *Subject:* Re: [Servingxml-help] Flat file structure validation > > > > > > > > Thanks Daniel. This is helpful. > > > > Another question on the same file structure. > > > > > > > > It looks like serving xml processes one record (using onRecord) at a > > > > time. Can I have control over 2 record at a time while converting flat file > > > > to xml. I want to convert the below flat file as follows. Essentially I > > > > want to put one field from E record and one field from S record into > > > > <group1>. So while processing E record, I need information from the next > > > > record also. > > > > > > > > E1001FIELD0 > > > > S1001FIELD1FIELD2 > > > > E1002FIELDA > > > > S1002FIELDBFIELDC > > > > > > > > TO > > > > > > > > <record fileno=1001> > > > > <group1> > > > > <a>FIELD0</a> > > > > <b>FIELD2</b> > > > > </group1> > > > > <group2> > > > > <c>FIELD1</c> > > > > </group2> > > > > </record> > > > > > > > > > > > > > > > > > > > > On 3/12/07, Daniel Parker <dan...@sy...> wrote: > > > > > > > > > > Hello Ravi, > > > > > > > > > > I think you'd have to transform the flat file to XML and express > > > > > the rules in an XML Schema, using the msv:schemaValidator (previously > > > > > msv:msvFilter) to validate the XML. Alternatively, you could write a custom > > > > > record filter in Java, and buffer enough records until you could determine > > > > > whether they were valid, and discard them or throw an exception if you find > > > > > an error. But other than that, the tool doesn't provide anything special > > > > > for validating groups of records, as opposed to individual records. > > > > > > > > > > Best regards, > > > > > Daniel > > > > > > > > > > ----- Original Message ----- > > > > > *From:* Ravikumar Tadysetty <ra...@gm...> > > > > > *To:* ser...@li... > > > > > *Sent:* Friday, March 09, 2007 12:16 AM > > > > > *Subject:* [Servingxml-help] Flat file structure validation > > > > > > > > > > Hi > > > > > > > > > > I am trying to use servingxml to validate and transform flat files > > > > > to xml. I am wondering if this toolkit can help validate the flat file which > > > > > looks as follows: > > > > > > > > > > Every row is a record with position #1 as record type. Position > > > > > #2-5 as filenumber and the rest as misc fields. I want to validate that for > > > > > every filenumber, there exists atleast 1 E record followed by 0 or more S > > > > > records. > > > > > > > > > > E1001FIELD0 > > > > > S1001FIELD1FIELD2 > > > > > S1001FIELD3FIELD4 > > > > > E1002FIELDA > > > > > S1002FIELDBFIELDC > > > > > > > > > > Thanks > > > > > Ravi > > > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > > > > share your > > > > > opinions on IT & business topics through brief surveys-and earn > > > > > cash > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > > > > > > > > ------------------------------ > > > > > > > > > > _______________________________________________ > > > > > Servingxml-help mailing list > > > > > Ser...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/servingxml-help > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > Join SourceForge.net's Techsay panel and you'll get the chance to > > > > share your > > > > opinions on IT & business topics through brief surveys-and earn cash > > > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > _______________________________________________ > > > > Servingxml-help mailing list > > > > Ser...@li... > > > > https://lists.sourceforge.net/lists/listinfo/servingxml-help > > > > > > > > > > > > > > |