Are there any plans for publishing NIEM 3.0 dictionaries?
I am trying to build the dictionaries from the XSDs (starting with core) and not getting anywhere quick. I've been getting heap errors that I keep increasing the memory. The console seems to be always outputting comments related to InternationalCode, Locations, Dates. I am sure if this is correct or part of some recursion loop.
Perhaps, I am going about the process incorrectly, followed the YouTube videos, and if the proper procedures could be posted that might be beneficial.
We would like to try CAM with NIEM 3.0 and any advice we could get would be greatly appreciated.
Thanks!
Dave
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Apologies for the delay in us getting to this. As you are finding out its not an easy task - due to hidden issues in the NIEM schema themselves.
You are correct though - starting with NIEM core is best. There's two things you need to know.
edit structures.xsd - and comment out all those unnecessary attributes - that vastly (5x) reduces size of everything (e.g. metadata, ref-ids et al).
NIEM core contains recursive elements (that's all those messages you are seeing in the import console, over and over and over). Its a slow laborious process to label these (since NIEM does not identify them for you). Basically you have to look at the console log - figure out the recursion, and then go label the head of the loop - with an annotation (add it in the NIEM core schema) on the type - with the simple word - recursive - e.g. ones I know already are organization and location - both reference themselves. With the recursive annotation in place - the CAM import will not loop on that component.
If you have time to fight through that - fantastic! Otherwise - we are working on a CAM v3.2 release right now - bug fixes to some important XSLTs - and once we have that out - then NIEM 3.0 is our next priority. Estimate - it is going to be a couple more weeks till we start on it - and probably a week to get NIEM-core itself done - and then after that will publish each other domain dictionary once they are successfully loaded.
Hope that is helpful,
David
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just to jump on the tail of this discussion. We're evaluating NIEM 3.1 (beta) and would like to use CAM in this process. So we need to import the 3.1 schemas.
I'm starting with NIEM core and run into a problem pretty quickly with the first element. ActivityConveyanceAssociationType gets stuck in a loop dealing with it's parent element: AssociationType (actually the parent of nc:AssocationType, structures:AssociationType is the issue)
It looks like a recursion problem with this being repeatedly output to the console:
--- complexType AssociationType not matched for ns http://release.niem.gov/niem/structures/3.0/ [info]extension with @base content: structures:AssociationType [info]xsd:complexType[xsd:complexContent]:AssociationType extn:structures:AssociationType; ns[] [info]Applying xsd:complexContent[complex]:;[] [info]extension (complex): structures:AssociationType - AssociationType [info]extension lookup base type: AssociationType [info]applying type for name: AssociationType
As this is an inherited element, I'm not sure how to go about marking it as recursive - commenting out the extension allows the parsing to continue, so I figure there isn't an issue with importing that structures file.
Any pointers in figuring this out? I've followed the instructions above in commenting out the attributes in the structures.xsd file.
Any help, greatly appreciated.
tom
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I will download the 3.1 package and take a look too. Not sure what is going on there with AssociationType - usually CAM can spot automatically those direct recursive references.
To manually label a schema component that is causing recursion and looping - you need to add:
Tom, update on this. I just tried the niem-core.xsd with CAM 3.2 - and it crashes.
I have CAM 3.2.1 with several important fixes for xsd extensions - and it is working. I need to focus on getting that release out the door here - we're in the middle of regression testing right now.
Make no mistake - this NIEM 3.0/3.1 is a big challenge. I'm already seeing major raft of recursive references around Organization and Person. CAM is spotting the direct recursion - its the indirect that has to be manually labelled.
It's deja vu sadly - as NIEM 2.0 was similar. Unfortunately NIEM 3.0 is very different - so the patches that worked to make NIEM 2.0 usable are not simply transferable.
OK - bottom line - let's get CAM v3.2.1 shipped - then we can focus on NIEM 3.0 importing next week...
I think if we can crack the recursive references between Person, Organization and Location it will go a long way to making things work - that was the experience the last time. Sadly there is no easy way to catch these with simple search.
Thanks, David
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Goutham, let me take a look in the morning. There is now a new beta on a NIEM/JSON tool - and obviously JSON does not support recursion - so let me see if that can unlock this for us... more shortly. David
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks David. Please let us know if the beta tool is availble for download or if any patches are availble for camed to resolve the issue with recursion.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I tried the JSON path and it is a deadend - no help there. The problem with recursion is not with the CAM import - it detects direct recursion - the problem is indirect recursion - and that's systemic to NIEM XSD. Basically the only way out is to manually label those in the XSD - and we know where the obvious ones are - Person references Address references Organization references Person and then Organization references Location references Address. Those are the top 2 - there are more.
I noted that request above for Biometric domain. That is small schema - might be good place to start - importing that. Of course it pulls in NIEM core which is where the issues are. However - if you comment out a big enough chunk (imports) - likely you will avoid enough of the recursion to be able to get a subset loading. Trial and error is the only way - and then once we have that set of comments added that label where the locations for recursion are - all good. The problem is - when I did this for NIEM 2.0 - they then changed everything for NIEM 3.0 so those old ones not reusable.
Last edit: drrwebber 2017-10-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Are there any plans for publishing NIEM 3.0 dictionaries?
I am trying to build the dictionaries from the XSDs (starting with core) and not getting anywhere quick. I've been getting heap errors that I keep increasing the memory. The console seems to be always outputting comments related to InternationalCode, Locations, Dates. I am sure if this is correct or part of some recursion loop.
Perhaps, I am going about the process incorrectly, followed the YouTube videos, and if the proper procedures could be posted that might be beneficial.
We would like to try CAM with NIEM 3.0 and any advice we could get would be greatly appreciated.
Thanks!
Dave
Apologies for the delay in us getting to this. As you are finding out its not an easy task - due to hidden issues in the NIEM schema themselves.
You are correct though - starting with NIEM core is best. There's two things you need to know.
edit structures.xsd - and comment out all those unnecessary attributes - that vastly (5x) reduces size of everything (e.g. metadata, ref-ids et al).
NIEM core contains recursive elements (that's all those messages you are seeing in the import console, over and over and over). Its a slow laborious process to label these (since NIEM does not identify them for you). Basically you have to look at the console log - figure out the recursion, and then go label the head of the loop - with an annotation (add it in the NIEM core schema) on the type - with the simple word - recursive - e.g. ones I know already are organization and location - both reference themselves. With the recursive annotation in place - the CAM import will not loop on that component.
If you have time to fight through that - fantastic! Otherwise - we are working on a CAM v3.2 release right now - bug fixes to some important XSLTs - and once we have that out - then NIEM 3.0 is our next priority. Estimate - it is going to be a couple more weeks till we start on it - and probably a week to get NIEM-core itself done - and then after that will publish each other domain dictionary once they are successfully loaded.
Hope that is helpful,
David
Hi,
We are also interested in NIEM dictionaries, especially on biometrics domain.
Any news on this?
Thank you.
Cristian
Just to jump on the tail of this discussion. We're evaluating NIEM 3.1 (beta) and would like to use CAM in this process. So we need to import the 3.1 schemas.
I'm starting with NIEM core and run into a problem pretty quickly with the first element. ActivityConveyanceAssociationType gets stuck in a loop dealing with it's parent element: AssociationType (actually the parent of nc:AssocationType, structures:AssociationType is the issue)
It looks like a recursion problem with this being repeatedly output to the console:
As this is an inherited element, I'm not sure how to go about marking it as recursive - commenting out the extension allows the parsing to continue, so I figure there isn't an issue with importing that structures file.
Any pointers in figuring this out? I've followed the instructions above in commenting out the attributes in the structures.xsd file.
Any help, greatly appreciated.
tom
Tom,
I will download the 3.1 package and take a look too. Not sure what is going on there with AssociationType - usually CAM can spot automatically those direct recursive references.
To manually label a schema component that is causing recursion and looping - you need to add:
to it. Once that is labelled - processing should only reference it once.
Tom, update on this. I just tried the niem-core.xsd with CAM 3.2 - and it crashes.
I have CAM 3.2.1 with several important fixes for xsd extensions - and it is working. I need to focus on getting that release out the door here - we're in the middle of regression testing right now.
Make no mistake - this NIEM 3.0/3.1 is a big challenge. I'm already seeing major raft of recursive references around Organization and Person. CAM is spotting the direct recursion - its the indirect that has to be manually labelled.
It's deja vu sadly - as NIEM 2.0 was similar. Unfortunately NIEM 3.0 is very different - so the patches that worked to make NIEM 2.0 usable are not simply transferable.
OK - bottom line - let's get CAM v3.2.1 shipped - then we can focus on NIEM 3.0 importing next week...
I think if we can crack the recursive references between Person, Organization and Location it will go a long way to making things work - that was the experience the last time. Sadly there is no easy way to catch these with simple search.
Thanks, David
Hello David,
Has NIEM 3.0 been imported yet? We are still running into recursive reference issues with Person object and were wondering if this has been fixed yet.
Thanks David,
I appreciate the effort.
tom
Goutham, let me take a look in the morning. There is now a new beta on a NIEM/JSON tool - and obviously JSON does not support recursion - so let me see if that can unlock this for us... more shortly. David
Thanks David. Please let us know if the beta tool is availble for download or if any patches are availble for camed to resolve the issue with recursion.
Hello David,
Can you please give us an update on this?
I tried the JSON path and it is a deadend - no help there. The problem with recursion is not with the CAM import - it detects direct recursion - the problem is indirect recursion - and that's systemic to NIEM XSD. Basically the only way out is to manually label those in the XSD - and we know where the obvious ones are - Person references Address references Organization references Person and then Organization references Location references Address. Those are the top 2 - there are more.
I noted that request above for Biometric domain. That is small schema - might be good place to start - importing that. Of course it pulls in NIEM core which is where the issues are. However - if you comment out a big enough chunk (imports) - likely you will avoid enough of the recursion to be able to get a subset loading. Trial and error is the only way - and then once we have that set of comments added that label where the locations for recursion are - all good. The problem is - when I did this for NIEM 2.0 - they then changed everything for NIEM 3.0 so those old ones not reusable.
Last edit: drrwebber 2017-10-07