#4 XS:Imports Namespace lost

[Robert F Lario]

It appears that the integrity of the namespaces are not maintained per the definition of xs:import.

I believe that when using xs:import, the namespace of the imported XSD is maintained within the host xsd that is importing. It appears that xs:import is handled like an xs:include in the code, which is incorrect.

so, given:

1) 'AFile.xsd' with element 'A1' typed as a complexType 'A1c' and a namespace of 'An'

2) 'BFile.xsd' with element 'B1' typed as complexType B1c and a namespace of 'Bn' that imports AFile.xsd

Per the spec BFile should have :
An:A1
An:A1c
Bn:B1
Bn:B1c

But the code shows :
Bn:A1
Bn:A1c
Bn:B1
BnB1c

Note this code :
In 'process_includes.py' note function 'clear_includes_and_imports(node)'
function treats xs:includes and xs:imports both as an xs:include

[Dave Kuhlman]

Robert,

Not too sure, but I do not believe I have yet been able to reproduce
your problem.

I've attached files that I created and used for my tests.

When I test with these files, here is what I did and what I see:

$ ./generateDS.py -o testmod.py bfile.xsd
$ python testmod.py test01.xml
<?xml version="1.0" ?>
<bn:B1 xmlns:bn="http://example.org/namespaceB">
    <fieldB1>sample one</fieldB1>
    <fieldB2>456</fieldB2>
    <fieldB3>
        <fieldA1>sample two</fieldA1>
        <fieldA2>123</fieldA2>
    </fieldB3>
</bn:B1>

That seems right, but maybe not what you are trying to report.

Can you tell me how I need to modify my test files in order to
reproduce the issue you are reporting. Thanks for any help.

And, by the way, I agree with you that there must be something amiss
where process_includes.py is treating xs:include and xs:import
the same way. A bit of excuse making -- that code was written years
ago when I understood XML name spaces even more poorly than I do now.

Dave

[Robert F Lario]

Hi Dave,

The approach appears to walk recursively across the imports and includes and flatten into one tree. The problem is how the flattening is done. It appears that imports are treated in the same manner as includes. Per the spec, included constructs use the host xsd namespace. However, imports should maintain their native namespaces in the host xsd.

This becomes an issue when you have two or more elements with the same name in two or more imported xsds. I end up with collisions. Also a problem when generating code. I would want to generate code for each construct with their respective namespace.

To see the issue added code (in prep_schema) at line 38-41 "# ADD THIS THIS ***"

1. def prep_schema_doc(infile, outfile, inpath, options):

2. Note: infile has been opened in binary mode.

3. if inpath.startswith('/'):
4. inpath = os.path.relpath(inpath)
5. doc1 = etree.parse(infile)
6. root1 = doc1.getroot()
7. params = Params()
8. params.parent_url = infile
9. params.base_url = os.path.split(inpath)[0]
10. inserts = []
11. ns_dict = {}
12. schema_ns_dict = {}
13. rename_data = RenameData()
14. schema_ns_dict.update(root1.nsmap)
15. if not options.no_collect_includes:
16. collect_inserts(root1, params, inserts, ns_dict,
17. schema_ns_dict, rename_data, options)
18. make_names_unique(root1, rename_data, options)
19. fixup_refs(root1, inserts, rename_data)
20. fixup_refs(root1, root1.getchildren(), rename_data)
21. root2 = copy.copy(root1)
22. clear_includes_and_imports(root2)
23. for insert_node in inserts:
24. root2.append(insert_node)
25. else:
26. root2 = root1
27. if not options.no_redefine_groups:
28. process_groups(root2)
29. raise_anon_complextypes(root2, rename_data)
30. fix_type_names(root2, options)
31. doc2 = etree.ElementTree(root2)
32. sm = 3
33. sm = sys.version_info.major
34. if sm == 2:
35. doc2.write(outfile)
36. else:
    37.

37. ADD THIS THIS ***

38. file = open(r"C:\junk\testrfl.xsd", 'w')
39. file.write(etree.tostring(root2,inclusive_ns_prefixes=True).decode('utf-8'))
40. file.close()

41. *****

    
    43.
42. outfile.write(etree.tostring(root2,inclusive_ns_prefixes=True).decode('utf-8'))

43. dbg

44. print('\nmapping:')

45. for item in rename_data.name_mappings.items():

46. print(' {}'.format(item))

47. print('\n')

48. return doc2, ns_dict, schema_ns_dict, rename_data

49. end prep_schema_doc

This is the flatten xsd :

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:an="http://example.org/namespaceA" xmlns:bn="http://example.org/namespaceB" targetnamespace="http://example.org/namespaceB">

<xs:element name="B1" type="bn:B1c">
<xs:element name="AB1" type="bn:A1c">
<xs:complextype name="A1c1">
<xs:sequence>
<xs:element name="fieldaa1" type="xs:string">
<xs:element name="fieldaa2" type="xs:integer">
<xs:element name="fieldaa3" type="an:A1c">
</xs:element></xs:element></xs:element></xs:sequence>
</xs:complextype>
<xs:complextype name="B1c">
<xs:sequence>
<xs:element name="fieldB1" type="xs:string">
<xs:element name="fieldB2" type="xs:integer">
<xs:element name="fieldB3" type="an:A1c">
</xs:element></xs:element></xs:element></xs:sequence>
</xs:complextype></xs:element></xs:element></xs:schema>

<xs:element name="A1" type="an:A1c">

<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" type="xs:string">
<xs:element name="fieldA2" type="xs:integer">
</xs:element></xs:element></xs:sequence>
</xs:complextype></xs:element>

Note :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" type="xs:string">
<xs:element name="fieldA2" type="xs:integer">
</xs:element></xs:element></xs:sequence>
</xs:complextype>
Is not scoped by http://example.org/namespaceA

Also note :
<xs:element name="AB1" type="bn:A1c"></xs:element>

AB1 is now defined as :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" type="xs:string">
<xs:element name="fieldA2" type="xs:integer">
</xs:element></xs:element></xs:sequence>
</xs:complextype>

But was originally defined as :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldaa1" type="xs:string">
<xs:element name="fieldaa2" type="xs:integer">
<xs:element name="fieldaa3" type="an:A1c">
</xs:element></xs:element></xs:element></xs:sequence>
</xs:complextype>
From: Dave Kuhlman dkuhlman@users.sourceforge.net
Sent: Monday, September 14, 2020 6:30 PM
To: [generateds:tickets] 4@tickets.generateds.p.re.sourceforge.net
Subject: [generateds:tickets] #4 XS:Imports Namespace lost

Robert,

Not too sure, but I do not believe I have yet been able to reproduce
your problem.

I've attached files that I created and used for my tests.

When I test with these files, here is what I did and what I see:

$ ./generateDS.py -o testmod.py bfile.xsd

$ python testmod.py test01.xml

<bn:b1 xmlns:bn="http://example.org/namespaceB"></bn:b1>

<fieldB1>sample one</fieldB1>

<fieldB2>456</fieldB2>

<fieldB3>

    <fieldA1>sample two</fieldA1>

    <fieldA2>123</fieldA2>

</fieldB3>

That seems right, but maybe not what you are trying to report.

Can you tell me how I need to modify my test files in order to
reproduce the issue you are reporting. Thanks for any help.

And, by the way, I agree with you that there must be something amiss
where process_includes.py is treating xs:include and xs:import
the same way. A bit of excuse making -- that code was written years
ago when I understood XML name spaces even more poorly than I do now.

Dave

Attachments:

• test_files.ziphttps://sourceforge.net/p/generateds/tickets/_discuss/thread/7b54d593f5/6dce/attachment/test_files.zip (1.1 kB; application/zip)

---

[tickets:#4]https://sourceforge.net/p/generateds/tickets/4/ XS:Imports Namespace lost

Status: open
Milestone: 1.0
Created: Sun Sep 13, 2020 05:37 PM UTC by Robert F Lario
Last Updated: Sun Sep 13, 2020 05:37 PM UTC
Owner: Dave Kuhlman

It appears that the integrity of the namespaces are not maintained per the definition of xs:import.

I believe that when using xs:import, the namespace of the imported XSD is maintained within the host xsd that is importing. It appears that xs:import is handled like an xs:include in the code, which is incorrect.

so, given:

1) 'AFile.xsd' with element 'A1' typed as a complexType 'A1c' and a namespace of 'An'

2) 'BFile.xsd' with element 'B1' typed as complexType B1c and a namespace of 'Bn' that imports AFile.xsd

Per the spec BFile should have :
An:A1
An:A1c
Bn:B1
Bn:B1c

But the code shows :
Bn:A1
Bn:A1c
Bn:B1
BnB1c

Note this code :
In 'process_includes.py' note function 'clear_includes_and_imports(node)'
function treats xs:includes and xs:imports both as an xs:include

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/generateds/tickets/4/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[Dave Kuhlman]

Robert,

Looking at this again.

I generated a module with this:

 $ generateDS.py -f -o tmp01sup.py -s tmp01sub.py --super tmp01sup

--member-specs=dict --export="write etree validate" bfile.xsd

I suspect that you are right, and that there is a problem here.
Perhaps namespaces and namespace prefixes will not be produced
correctly during export.

However, when I generate a module from the modified bfile.xsd and
afile.xsd. I do not get name collisions. I see that we generate
classes A1c, A1c1, and B1c. As you mention below, class
A1c1, in this case, is the complex type defined in schema
bfile.xsd. It has been renamed in order to avoid use of a
duplicate name. One of the reasons for this renaming is that we
must generate all these classes in a single Python namespace.

Class A1c1 has a member variable that contains instances of class
A1c (the one defined in namespace an).

Near the bottom of the generated module, you will find something
like this:

 RenameMappings_ = {
     "{http://example.org/namespaceB}A1c": "A1c1",
 }

This is intended to tell you what gets renamed to what. For one, it
could help you determine that, if you want to create an instance of
your new added element AB1, you need to create an instance of
class A1c1, which is the renamed one from namespace
http://example.org/namespaceB.

So, that renaming, again that you refer in your message, is
intended. It may not be ideal, but it's the best that
generateDS.py can do with duplicate names in multiple namespaces.
And, actually, it took me a long time to implement something that
was even that good (or less bad).

Does that help? Please let me know if your think there is still
something wrong, perhaps something I've missed?

Dave

On Tue 15 Sep 2020 12:38:18 PM PDT, Robert F Lario wrote:

Hi Dave,

The approach appears to walk recursively across the imports and
includes and flatten into one tree. The problem is how the
flattening is done. It appears that imports are treated in the same
manner as includes. Per the spec, included constructs use the host
xsd namespace. However, imports should maintain their native
namespaces in the host xsd.

This becomes an issue when you have two or more elements with the
same name in two or more imported xsds. I end up with collisions.
Also a problem when generating code. I would want to generate code
for each construct with their respective namespace.

To see the issue added code (in prep_schema) at line 38-41 "# ADD
THIS THIS ***"

1. def prep_schema_doc(infile, outfile, inpath, options):

2. Note: infile has been opened in binary mode.

3. if inpath.startswith('/'):
4. inpath = os.path.relpath(inpath)
5. doc1 = etree.parse(infile)
6. root1 = doc1.getroot()
7. params = Params()
8. params.parent_url = infile
9. params.base_url = os.path.split(inpath)[0]
10. inserts = []
11. ns_dict = {}
12. schema_ns_dict = {}
13. rename_data = RenameData()
14. schema_ns_dict.update(root1.nsmap)
15. if not options.no_collect_includes:
16. collect_inserts(root1, params, inserts, ns_dict,
17. schema_ns_dict, rename_data, options)
18. make_names_unique(root1, rename_data, options)
19. fixup_refs(root1, inserts, rename_data)
20. fixup_refs(root1, root1.getchildren(), rename_data)
21. root2 = copy.copy(root1)
22. clear_includes_and_imports(root2)
23. for insert_node in inserts:
24. root2.append(insert_node)
25. else:
26. root2 = root1
27. if not options.no_redefine_groups:
28. process_groups(root2)
29. raise_anon_complextypes(root2, rename_data)
30. fix_type_names(root2, options)
31. doc2 = etree.ElementTree(root2)
32. sm = 3
33. sm = sys.version_info.major
34. if sm == 2:
35. doc2.write(outfile)
36. else:
    37.

37. ADD THIS THIS ***

38. file = open(r"C:\junk\testrfl.xsd", 'w')
39. file.write(etree.tostring(root2,inclusive_ns_prefixes=True).decode('utf-8'))
40. file.close()

41. *****

    
    43.
42. outfile.write(etree.tostring(root2,inclusive_ns_prefixes=True).decode('utf-8'))

43. dbg

44. print('\nmapping:')

45. for item in rename_data.name_mappings.items():

46. print(' {}'.format(item))

47. print('\n')

48. return doc2, ns_dict, schema_ns_dict, rename_data

49. end prep_schema_doc

This is the flatten xsd :

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" <br="">xmlns:an="http://example.org/namespaceA"
xmlns:bn="http://example.org/namespaceB"
targetNamespace="http://example.org/namespaceB">

<xs:element name="B1" type="bn:B1c">
<xs:element name="AB1" type="bn:A1c">
<xs:complextype name="A1c1">
<xs:sequence>
<xs:element<br>name="fieldaa1" type="xs:string"/>
<xs:element<br>name="fieldaa2" type="xs:integer"/>
<xs:element<br>name="fieldaa3" type="an:A1c"/>
</xs:element<br></xs:element<br></xs:element<br></xs:sequence>
</xs:complextype>
<xs:complextype name="B1c">
<xs:sequence>
<xs:element name="fieldB1" <br="">type="xs:string"/>
<xs:element name="fieldB2" <br="">type="xs:integer"/>
<xs:element name="fieldB3" <br="">type="an:A1c"/>
</xs:element></xs:element></xs:element></xs:sequence>
</xs:complextype>

<xs:element name="A1" type="an:A1c">

<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" <br="">type="xs:string"/>
<xs:element name="fieldA2" <br="">type="xs:integer"/>
</xs:element></xs:element></xs:sequence>
</xs:complextype>
</xs:element></xs:element></xs:element></xs:schema>

Note :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" <br="">type="xs:string"/>
<xs:element name="fieldA2" <br="">type="xs:integer"/>
</xs:element></xs:element></xs:sequence>
</xs:complextype>
Is not scoped by http://example.org/namespaceA

Also note :
<xs:element name="AB1" type="bn:A1c"></xs:element>

AB1 is now defined as :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element name="fieldA1" <br="">type="xs:string"/>
<xs:element name="fieldA2" <br="">type="xs:integer"/>
</xs:element></xs:element></xs:sequence>
</xs:complextype>

But was originally defined as :
<xs:complextype name="A1c">
<xs:sequence>
<xs:element<br>name="fieldaa1" type="xs:string"/>
<xs:element<br>name="fieldaa2" type="xs:integer"/>
<xs:element<br>name="fieldaa3" type="an:A1c"/>
</xs:element<br></xs:element<br></xs:element<br></xs:sequence>
</xs:complextype>
From: Dave Kuhlman dkuhlman@users.sourceforge.net
Sent: Monday, September 14, 2020 6:30 PM
To: [generateds:tickets] 4@tickets.generateds.p.re.sourceforge.net
Subject: [generateds:tickets] #4 XS:Imports Namespace lost

Robert,

Not too sure, but I do not believe I have yet been able to reproduce
your problem.

I've attached files that I created and used for my tests.

When I test with these files, here is what I did and what I see:

$ ./generateDS.py -o testmod.py bfile.xsd

$ python testmod.py test01.xml

<bn:b1 xmlns:bn="http://example.org/namespaceB"></bn:b1>

<fieldB1>sample one</fieldB1>

<fieldB2>456</fieldB2>

<fieldB3>

    <fieldA1>sample two</fieldA1>

    <fieldA2>123</fieldA2>

</fieldB3>

That seems right, but maybe not what you are trying to report.

Can you tell me how I need to modify my test files in order to
reproduce the issue you are reporting. Thanks for any help.

And, by the way, I agree with you that there must be something amiss
where process_includes.py is treating xs:include and xs:import
the same way. A bit of excuse making -- that code was written years
ago when I understood XML name spaces even more poorly than I do now.

Dave

Attachments:

• test_files.ziphttps://sourceforge.net/p/generateds/tickets/_discuss/thread/7b54d593f5/6dce/attachment/test_files.zip (1.1 kB;
  application/zip)

---

[tickets:#4]https://sourceforge.net/p/generateds/tickets/4/
XS:Imports Namespace lost

Status: open
Milestone: 1.0
Created: Sun Sep 13, 2020 05:37 PM UTC by Robert F Lario
Last Updated: Sun Sep 13, 2020 05:37 PM UTC
Owner: Dave Kuhlman

It appears that the integrity of the namespaces are not maintained
per the definition of xs:import.

I believe that when using xs:import, the namespace of the imported
XSD is maintained within the host xsd that is importing. It appears
that xs:import is handled like an xs:include in the code, which is
incorrect.

so, given:

1) 'AFile.xsd' with element 'A1' typed as a complexType 'A1c' and a
namespace of 'An'

2) 'BFile.xsd' with element 'B1' typed as complexType B1c and a
namespace of 'Bn' that imports AFile.xsd

Per the spec BFile should have :
An:A1
An:A1c
Bn:B1
Bn:B1c

But the code shows :
Bn:A1
Bn:A1c
Bn:B1
BnB1c

Note this code :
In 'process_includes.py' note function 'clear_includes_and_imports(node)'
function treats xs:includes and xs:imports both as an xs:include

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/generateds/tickets/4/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

---

** [tickets:#4] XS:Imports Namespace lost**

Status: open
Milestone: 1.0
Created: Sun Sep 13, 2020 05:37 PM UTC by Robert F Lario
Last Updated: Mon Sep 14, 2020 10:29 PM UTC
Owner: Dave Kuhlman

It appears that the integrity of the namespaces are not maintained
per the definition of xs:import.

I believe that when using xs:import, the namespace of the imported
XSD is maintained within the host xsd that is importing. It appears
that xs:import is handled like an xs:include in the code, which is
incorrect.

so, given:

1) 'AFile.xsd' with element 'A1' typed as a complexType 'A1c' and a
namespace of 'An'

2) 'BFile.xsd' with element 'B1' typed as complexType B1c and a
namespace of 'Bn' that imports AFile.xsd

Per the spec BFile should have :
An:A1
An:A1c
Bn:B1
Bn:B1c

But the code shows :
Bn:A1
Bn:A1c
Bn:B1
BnB1c

Note this code :
In 'process_includes.py' note function 'clear_includes_and_imports(node)'
function treats xs:includes and xs:imports both as an xs:include

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/generateds/tickets/4/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

--
Dave Kuhlman
http://www.davekuhlman.org

[Dave Kuhlman]

Robert,

Looking at this again.

I generated a module with this:

$ generateDS.py -f -o tmp01sup.py -s tmp01sub.py --super tmp01sup --member-specs=dict --export="write etree validate" bfile.xsd

I suspect that you are right, and that there is a problem here.
Perhaps namespaces and namespace prefixes will not be produced
correctly during export.

However, when I generate a module from the modified bfile.xsd and
afile.xsd. I do not get name collisions. I see that we generate
classes A1c, A1c1, and B1c. As you mention below, class
A1c1, in this case, is the complex type defined in schema
bfile.xsd. It has been renamed in order to avoid use of a
duplicate name. One of the reasons for this renaming is that we
must generate all these classes in a single Python namespace.

Class A1c1 has a member variable that contains instances of class
A1c (the one defined in namespace an).

Near the bottom of the generated module, you will find something
like this:

RenameMappings_ = {
    "{http://example.org/namespaceB}A1c": "A1c1",
}

This is intended to tell you what gets renamed to what. For one, it
could help you determine that, if you want to create an instance of
your new added element AB1, you need to create an instance of
class A1c1, which is the renamed one from namespace
http://example.org/namespaceB.

So, that renaming, again that you refer in your message, is
intended. It may not be ideal, but it's the best that
generateDS.py can do with duplicate names in multiple namespaces.
And, actually, it took me a long time to implement something that
was even that good (or less bad).

Does that help? Please let me know if your think there is still
something wrong, perhaps something I've missed?

Dave