Re: [Generateds-discuss] How to create/ generate new xml from generated API

[Dave K. <dku...@da...>]

Daniel,

Thanks for that offer of help with a tutorial.

There already is a tutorial, *but* it needs lots of work.
I have not looked at it for several years, at least.
The tutorial is hidden in the tutorial subdirectory, although you
will need to download the source distribution or clone the
repository from SourceForge:
https://sourceforge.net/projects/generateds/

Some needed changes that jump out at me:

- The section on the "The graphical front-end" should be deleted
   completely.  The graphical front end no longer can be run, now
   that there have been changes to the underlying graphics library
   (`gtk` I believe) and I was not willing to do all the work to make
   the necessary modifications.

- The section titled "Adding application-specific behavior", tells
   the user to generate a subclass module by using the "-s" command
   line option.  While that approach may be useful for some special
   purposes, I now feel that it's more straight-forward and more
   generally useful to generate a library module (with the "-o"
   command line option, then write user code in a separate module
   that imports the library module, and possibly copies and edits one
   of the "parse*" functions. So, maybe we need a new section on how
   to do that.

If you have ideas and suggestions, or if you want to work on this,
I'd be happy to discuss it.

And, I hope you are finding `generateDS` interesting, usable, and useful.

Dave


Quoting "Dewitte, Daniel W (US)" <dan...@ba...>:

> Dave,
>
> Thank you for the help, truthfully I have done very basic things  
> with Python up till about Feb.  Since then I have taught myself the  
> MVC programming model with PyQt5 packages.  As well as the package  
> installers and making the overall software OS agnostic.  I have 15+  
> years in LabView programming and a lot can be adopted but I haven’t  
> dealt with text based programming languages since highschool in  
> early 2000’s.
>
> Thank you for the compliments.  Maybe through all of this I can  
> assist you in writing an in-depth tutorial of writing an instance to  
> help others out.
>
> Sent with BlackBerry Work
> (www.blackberry.com)
>
>
> From: Dave Kuhlman  
> <dku...@da...<mailto:dku...@da...>>
> Date: Monday, Jun 06, 2022, 3:23 PM
> To: Dewitte, Daniel W (US)  
> <dan...@ba...<mailto:dan...@ba...>>
> Cc: Gen...@li...  
> <Gen...@li...<mailto:Gen...@li...>>
> Subject: Re: [Generateds-discuss] How to create/ generate new xml  
> from generated API
>
> External Email Alert
>
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested  
> to click on a link, decrypt/open an attachment, or enable macros.   
> For further information on how to spot phishing, access  
> “Cybersecurity OneSpace Page” and report phishing by clicking the  
> button “Report Phishing” on the Outlook toolbar.
>
>
> Daniel,
>
> What?  You don't seem like a Python newbie to me.
>
> About the error -- I believe that exception message is telling us
> that it wants a different type for the value of
> `testResults01.Personnel.set_SystemOperator`.
>
> Short version -- see attached file `test01.py`.
>
> Long attempted explanation -- see the following:
>
> In the XML schema (`TestResults.xsd`), search for "SystemOperator".
> You will find the following:
>
>      <xs:element name="SystemOperator" type="Person">
>          <xs:annotation>
>              <xs:documentation>Identification information for the
> operator of the ATE.</xs:documentation>
>          </xs:annotation>
>      </xs:element>
>
> The `type="Person"` tells us it want an instance of class `Person`.
> So, look at the definition of class `Person` in the module that you
> generated with `generateDS.py` (`atdsApi.py`, I believe).
>
> We have to construct a instance of class Person.  And, since class
> `Person` contains an `Address`, we can construct an instance of
> class `Address`.
>
> I've attached a file (`test01.py`) with a modified version of your  
> code.  And,
> here is a snippet
>
>      mailingaddress = lib.MailingAddress(
>          Address1="123 Pretty Lane",
>          City="Bigcity",
>          State="Arizona",
>          Country="USA",
>          PostalCode="98765",
>      )
>      sysOp = lib.Person(
>          name="Dan",
>          Address=mailingaddress,
>          email="da...@so...",
>          phone="123.456.7890",
>          # etc
>          )
>      testResults01.Personnel.set_SystemOperator(sysOp)
>      print (testResults01.Personnel.get_SystemOperator())
>      testResults01.export(sys.stdout, 0)
>
> You can get clues about this if you look in the `build`,
> `_buildAttributes`, and `_buildChildren` methods in classes
> `PersonnelType`, `Person`, `Operator`, etc.  Notice that, for the
> `SystemOperator` member, it creates and instance of class `Person`.
> And, since class `Person` is a subclass of class `Operator`, you may
> want to look at the implementation of class `Operator` in
> `atdsApi.py`.
>
> It's a bit confusing.  It is to me at least.  But, then, I suppose
> XML and XML schema is supposed to be.  Sigh.
>
> Hope this helps.
>
> Dave
>
>
>
> Quoting "Dewitte, Daniel W (US)" <dan...@ba...>:
>
>> Hi Dave,
>>
>> Thank you for the pointers, yes this worked great for me.  I still
>> get a few errors when creating the API but I will worry about those
>> when I get to the external validator I have to use to ensure that
>> everything is working.
>>
>> I also wanted to thank you, I am new to Python with this being my
>> first major project.  I have played around with an RPi but nothing
>> major like up to this point.  So some of the syntax and ways that
>> Python uses is new to me and I am trying to learn.
>>
>> I do have some more questions at the moment and then I think I will
>> be more then off and running on this next phase of this project.
>> 1)  You have been able to show me on how to create the initial
>> instance of the Root Element, is that correct?  As long as the line
>> "testResults01 = lib.TestResults()" calls the root element class.
>> Is that correct?
>>
>> 2)  In the def __init__ of that element, I see that the input
>> arguments are the attributes and child elements of that root
>> element.  See attached image.
>> The first two in the init at the elements attributes, the rest are
>> child elements.  I can set the attributes by just replacing "None"
>> with their values (hardcoded or generated).  How to I generate the
>> child elements (i.e. "Personnel") so that I can start setting their
>> attributes and parameters?
>>
>> I have tried the following code:
>> import sys
>> import atdsApi as lib
>>
>> testResults01 = lib.TestResults(uuid = "1", securityClassification =
>> "Unclassified",
>>                                 Personnel = lib.PersonnelType())
>> print (testResults01.get_uuid())
>> print (testResults01.get_securityClassification())
>> testResults01.export(sys.stdout, 0)
>>
>> I get this result:
>> 1
>> Unclassified
>> <c:TestResults  xmlns:None="http://www.ieee.org/ATML/2006/Common"
>> xmlns:c="http://www.ieee.org/ATML/2006/Common"  uuid="1"
>> securityClassification="Unclassified">
>>     <c:Personnel/>
>> </c:TestResults>
>>
>> When I try to then go and set the attributes of Personnel which is
>> controlled by PersonnelType() I get an error.
>> Code:
>> import sys
>> import atdsApi as lib
>>
>> testResults01 = lib.TestResults(uuid = "1", securityClassification =
>> "Unclassified",
>>                                 Personnel = lib.PersonnelType())
>> print (testResults01.get_uuid())
>> print (testResults01.get_securityClassification())
>> testResults01.Personnel.set_SystemOperator("Dan")
>> print (testResults01.Personnel.get_SystemOperator())
>> testResults01.export(sys.stdout, 0)
>>
>> Error:
>> 1
>> Unclassified
>> Dan
>> <c:TestResults  xmlns:None="http://www.ieee.org/ATML/2006/Common"
>> xmlns:c="http://www.ieee.org/ATML/2006/Common"  uuid="1"
>> securityClassification="Unclassified">
>>     <c:Personnel>
>> Traceback (most recent call last):
>>
>>   File "H:\C4ISR S2 NPI\Projects\Voice of the Process\ATDS TDR
>> Importer - Python\ATDS File Creator\generateDS_xmlCreate.py", line
>> 16, in <module>
>>     testResults01.export(sys.stdout, 0)
>>
>>   File "H:\C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>> Importer - Python/ATDS File Creator\atdsApi.py", line 1314, in export
>>     self._exportChildren(outfile, level + 1, namespaceprefix_,
>> namespacedef_, name_='TestResults', pretty_print=pretty_print)
>>
>>   File "H:\C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>> Importer - Python/ATDS File Creator\atdsApi.py", line 1342, in
>> _exportChildren
>>     self.Personnel.export(outfile, level, namespaceprefix_,
>> namespacedef_='', name_='Personnel', pretty_print=pretty_print)
>>
>>   File "H:\C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>> Importer - Python/ATDS File Creator\atdsApi.py", line 10284, in export
>>     self._exportChildren(outfile, level + 1, namespaceprefix_,
>> namespacedef_, name_='PersonnelType', pretty_print=pretty_print)
>>
>>   File "H:\C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>> Importer - Python/ATDS File Creator\atdsApi.py", line 10304, in
>> _exportChildren
>>     self.SystemOperator.export(outfile, level, namespaceprefix_,
>> namespacedef_='', name_='SystemOperator', pretty_print=pretty_print)
>>
>> AttributeError: 'str' object has no attribute 'export'
>>
>> By reading the error it looks like it is actually on the export
>> function and not the set function as it is setting the System
>> Operator.
>> Any ideas?
>>
>> I am sure to have more questions soon.
>> Thank you again,
>>
>> Daniel DeWitte
>>
>> T:  +1-603-885-2602
>> M: +1-603-759-4476
>> E: dan...@ba...
>>
>> -----Original Message-----
>> From: Dave Kuhlman <dku...@da...>
>> Sent: Thursday, June 2, 2022 3:56 PM
>> To: Dewitte, Daniel W (US) <dan...@ba...>
>> Cc: Gen...@li...
>> Subject: Re: [Generateds-discuss] How to create/ generate new xml
>> from generated API
>>
>> External Email Alert
>>
>> This email has been sent from an account outside of the BAE Systems network.
>>
>> Please treat the email with caution, especially if you are requested
>> to click on a link, decrypt/open an attachment, or enable macros.
>> For further information on how to spot phishing, access
>> “Cybersecurity OneSpace Page” and report phishing by clicking the
>> button “Report Phishing” on the Outlook toolbar.
>>
>>
>> Daniel,
>>
>> Thanks for providing that info, command line, generated module, etc.
>>
>> When you run `generateDS.py`, the "-o" option means generate a main
>> module, and the "-s" option says generate a subclass module.  The
>> "--super=xxxx" tells `generateDS.py` to generate an `import`
>> statement in the subclass module that imports module `atdsApi.py`.
>> Since you have given the same name for "-o" and "-s" that module
>> imports itself.  And, maybe, the main module is over-written by the
>> subclass module.
>>
>> I don't know what your needs are, but it's likely that you do not
>> want to use either "-s" or "--super" command line options.  If you
>> do not need the subclass module, do not generate it.
>>
>> My example command line does not make this very clear because,
>> although it does use different names for "-o" and "-s", there is
>> only one letter difference.  Sorry for the confusion.
>>
>> Try running the following instead:
>>
>>      $ python generateDS.py -f -o atdsApi.py --member-specs=dict
>> --export="write etree validate" TestResults.xsd
>>
>> Then the following Python code should work:
>>
>>      import sys
>>      import atdsApi as lib
>>      testResults01 = lib.TestResult()
>>      testResults01.export(sys.stdout, 0)
>>
>> Notice that I added "sys." in front of "stdout".
>>
>> Hope this helps.  Let me know if/when you have more questions.
>>
>> Dave
>>
>>
>> Quoting "Dewitte, Daniel W (US)" <dan...@ba...>:
>>
>>> Hi Dave,
>>>
>>> Thank you for your quick response, so far the first part of what you
>>> showed me is working, I do get some warnings though.  It looks like
>>> because the TestResults.xsd names multiple namespace prefix's it is
>>> not carrying the "c:" prefix.
>>>
>>> 1)  H:\Python Training\GenerateDS_Docs>python generateDS.py -f -o
>>> atdsApi.py -s atdsApi.py --super=atdsApi --member-specs=dict
>>> --export="write etree validate" TestResults.xsd
>>> *** No suitable ns prefix for element "<XschemaElement name:
>>> "Outcome" type: "Outcome">".
>>> WARNING - H:\Python Training\GenerateDS_Docs\generateDS.py - _lookupNS
>>> - Unknown namespace prefix "c" used.
>>> WARNING - H:\Python Training\GenerateDS_Docs\generateDS.py - _lookupNS
>>> - Unknown namespace prefix "c" used.
>>> WARNING - H:\Python Training\GenerateDS_Docs\generateDS.py - _lookupNS
>>> - Unknown namespace prefix "c" used.
>>>
>>> 2)  I created a new .py file and have the following:
>>> # -*- coding: utf-8 -*-
>>> """
>>> Created on Wed May 25 11:02:45 2022
>>>
>>> @author: daniel.dewitte
>>> """
>>> import sys
>>> import atdsApi as lib
>>>
>>> testResults01 = lib.TestResult()
>>> testResults01.export(stdout, 0)
>>>
>>> When I try to run this .py I get the following errors:
>>> runfile('H:/C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>>> Importer - Python/ATDS File Creator/generateDS_xmlCreate.py',
>>> wdir='H:/C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR Importer
>>> - Python/ATDS File Creator') Traceback (most recent call last):
>>>
>>>   File "H:\C4ISR S2 NPI\Projects\Voice of the Process\ATDS TDR
>>> Importer - Python\ATDS File Creator\generateDS_xmlCreate.py", line 8,
>>> in <module>
>>>     import atdsApi as lib
>>>
>>>   File "H:\C4ISR S2 NPI/Projects/Voice of the Process/ATDS TDR
>>> Importer - Python/ATDS File Creator\atdsApi.py", line 68, in <module>
>>>     class TestResultsSub(supermod.TestResults):
>>>
>>> AttributeError: partially initialized module 'atdsApi' has no
>>> attribute 'TestResults' (most likely due to a circular import)
>>>
>>> Attached the api.py for reference.
>>>
>>> Thanks,
>>> Daniel DeWitte
>>>
>>> -----Original Message-----
>>> From: Dave Kuhlman <dku...@da...>
>>> Sent: Wednesday, June 1, 2022 7:08 PM
>>> To: Dewitte, Daniel W (US) <dan...@ba...>
>>> Cc: Gen...@li...
>>> Subject: Re: [Generateds-discuss] How to create/ generate new xml from
>>> generated API
>>>
>>> External Email Alert
>>>
>>> This email has been sent from an account outside of the BAE  
>>> Systems network.
>>>
>>> Please treat the email with caution, especially if you are requested
>>> to click on a link, decrypt/open an attachment, or enable macros.
>>> For further information on how to spot phishing, access “Cybersecurity
>>> OneSpace Page” and report phishing by clicking the button “Report
>>> Phishing” on the Outlook toolbar.
>>>
>>>
>>> Daniel,
>>>
>>> Thanks for taking the time to report this.
>>>
>>> Here is what I've tried so far:
>>>
>>> 1. I copied TestResults.xml and Common.xsd to a directory.
>>>
>>> 2. I ran the following command:
>>>
>>>         ./generateDS.py -f -o tmp01sup.py -s tmp01sub.py --super
>>> tmp01sup --member-specs=dict '--export=write etree validate'
>>> TestResults.xsd
>>>
>>>     That generated a module `tmp01sup.py`.
>>>
>>> 3. So, I started IPython (the plain Python interactive prompt would
>>>     also work) and did this:
>>>
>>>         [ins] In [1]: import tmp01sup as lib
>>>
>>>     Which also worked.
>>>
>>> 4. Then I created an instance of a class in the generated module:
>>>
>>>         [ins] In [5]: testresults01 = lib.TestResult()
>>>         [ins] In [6]: testresults01.export(sys.stdout, 0)
>>>         <c:TestResult
>>> xmlns:None="http://www.ieee.org/ATML/2006/Common"
>>> xmlns:c="http://www.ieee.org/ATML/2006/Common" />
>>>
>>>     Which also seemed to work.
>>>
>>> So, now I've got to read your message a bit more carefully so that I
>>> can figure out what is wrong for your case and how to fix it.
>>>
>>> I do not have the file `collect_schema_locations.py`, which you show
>>> in your message as generating an exception.  So, I can't test that.
>>> Did you generate that file with `generateDS.py`?
>>>
>>> However the exception "TypeError: empty namespace prefix is not
>>> supported in XPath" seems to occur when you have a key in the nsmap
>>> that is `None`.  Try removing it.  Then do the `xpath` search:
>>>
>>>      [ins] In [23]: nsmap = root.nsmap
>>>      [ins] In [24]: nsmap
>>>      Out[24]:
>>>      {'xs': 'http://www.w3.org/2001/XMLSchema',
>>>       'c': 'http://www.ieee.org/ATML/2006/Common',
>>>       None: 'http://www.ieee.org/ATML/2007/TestResults'}
>>>      [ins] In [25]: nsmap.pop(None)
>>>      Out[25]: 'http://www.ieee.org/ATML/2007/TestResults'
>>>      [ins] In [26]: nsmap
>>>      Out[26]:
>>>      {'xs': 'http://www.w3.org/2001/XMLSchema',
>>>       'c': 'http://www.ieee.org/ATML/2006/Common'}
>>>      [ins] In [27]: root.xpath('xs:import', namespaces=nsmap)
>>>      Out[27]: [<Element {http://www.w3.org/2001/XMLSchema}import at
>>> 0x7faf573acac0>]
>>>
>>> Does that help with your work?
>>>
>>> By the way, the "--one-file-per-xsd" capability is something that I
>>> do not understand, myself.  That code was contributed by someone else.
>>>
>>> Dave
>>>
>>>
>>> Quoting "Dewitte, Daniel W (US) via Generateds-discuss"
>>> <gen...@li...>:
>>>
>>>> Here are the actual files this time :)
>>>>
>>>> Daniel DeWitte
>>>>
>>>> From: Dewitte, Daniel W (US)
>>>> Sent: Wednesday, June 1, 2022 1:06 PM
>>>> To: 'Gen...@li...'
>>>> <Gen...@li...>
>>>> Subject: How to create/ generate new xml from generated API
>>>>
>>>> Hi Dave,
>>>>
>>>> This is one of the better tools that I have found and I think the only
>>>> one that can work the way I need it too, I just need some help with
>>>> getting to the finish line.
>>>>
>>>> My goal is generate a new .xml using a known schema .xsd with Python
>>>> as the base language (required).  I have 13 different reports that all
>>>> have a different format that are generated on a constant basis that I
>>>> need to import into a known .xsd format for upload to a database.
>>>>
>>>> I believe I either want to use the build methods, or the "one-per"
>>>> options, but I can't get either of them to work.  I asked about the
>>>> build method problem on stackover flow here:  Stackoverflow - Creating
>>>> a new XML from an XSD using GenerateDS and
>>>> Python<https://stackoverflow.com/questions/72397436/creating-a-new-xml
>>>> -from-an-xsd-using-generateds-and-python>
>>>>
>>>> But I am not sure that the build method is what I want to use.
>>>>
>>>> When I try the "one-per" I get the following errors:
>>>> Command line method, I believe this error is due to that the .xsd is
>>>> in a UTF-8 encoding.  How do I get this to work with UTF-8 encoding?
>>>>> (base) H:\Python Training\GenerateDS_Docs>python generateds.py
>>>>> --one-file-per-xsd --output-directory="OnePer"
>>>>> --module-suffix="One" TestResults.xsd Traceback (most recent call
>>>>> last):
>>>>> File "H:\Python Training\GenerateDS_Docs\generateds.py", line 9906,
>>>>> in <module>
>>>>     main()
>>>>> File "H:\Python Training\GenerateDS_Docs\generateds.py", line 9872,
>>>>> in main
>>>>     parseAndGenerate(
>>>>> File "H:\Python Training\GenerateDS_Docs\generateds.py", line 9318,
>>>>> in parseAndGenerate
>>>>     parser.parse(rootFile)
>>>>> File "C:\EngTools\Anaconda3\lib\xml\sax\expatreader.py", line 111, in
>>>>> parse
>>>>     xmlreader.IncrementalParser.parse(self, source)
>>>>> File "C:\EngTools\Anaconda3\lib\xml\sax\xmlreader.py", line 123, in
>>>>> parse
>>>>     buffer = file.read(self._bufsize)
>>>>> File "C:\EngTools\Anaconda3\lib\encodings\cp1252.py", line 23, in
>>>>> decode
>>>>     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
>>>>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in
>>>>> position 4013: character maps to <undefined>
>>>>
>>>> Approach 2 - Extraction and Generation Utilities:
>>>>> (base) h:\Python Training\GenerateDS_Docs\utils>python
>>>>> collect_schema_locations.py --force --verbose TestResults.xsd
>>>>> testJson.txt Traceback (most recent call last):
>>>>> File "h:\Python
>>>>> Training\GenerateDS_Docs\utils\collect_schema_locations.py", line
>>>>> 157, in <module>
>>>>     main()
>>>>> File "h:\Python
>>>>> Training\GenerateDS_Docs\utils\collect_schema_locations.py", line
>>>>> 148, in main
>>>>     extract_and_generate(infile, outfile, extract_locations_fn,
>>>> options)
>>>>> File "h:\Python
>>>>> Training\GenerateDS_Docs\utils\collect_schema_locations.py", line 86,
>>>>> in extract_and_generate
>>>>     locations = extract_locations(infile, options)
>>>>> File "h:\Python
>>>>> Training\GenerateDS_Docs\utils\collect_schema_locations.py", line 50,
>>>>> in extract_locations
>>>>     elements = root.xpath('xs:import', namespaces=root.nsmap)
>>>>> File "src/lxml/etree.pyx", line 1579, in lxml.etree._Element.xpath
>>>>> File "src/lxml/xpath.pxi", line 259, in
>>>>> lxml.etree.XPathElementEvaluator.__init__
>>>>> File "src/lxml/xpath.pxi", line 131, in
>>>>> lxml.etree._XPathEvaluatorBase.__init__
>>>>> File "src/lxml/xpath.pxi", line 55, in
>>>>> lxml.etree._XPathContext.__init__ File "src/lxml/extensions.pxi",
>>>>> line 81, in lxml.etree._BaseContext.__init__
>>>>> TypeError: empty namespace prefix is not supported in XPath
>>>>
>>>> Attached are the .xsd that I am using.  TestResults.xsd references
>>>> Common.xsd.
>>>>
>>>> Any help on getting started with my overall goal will be much
>>>> appreciated.  I am looking to create a new XML from a known .xsd and
>>>> add in the elements and child elements and attributes as needed.
>>>>
>>>> Thank you for the help,
>>>> Daniel DeWitte
>>>
>>>
>>>
>>> --
>>> Dave Kuhlman
>>> http://www.davekuhlman.org
>>
>>
>>
>> --
>> Dave Kuhlman
>> http://www.davekuhlman.org
>
>
>
> --
> Dave Kuhlman
> http://www.davekuhlman.org



-- 
Dave Kuhlman
http://www.davekuhlman.org