NIEM 3.0 Dictionaries

djeverson
2014-05-24
2015-04-24
  • djeverson

    djeverson - 2014-05-24

    Hi,

    Are there any plans for publishing NIEM 3.0 dictionaries?

    I am trying to build the dictionaries from the XSDs (starting with core) and not getting anywhere quick. I've been getting heap errors that I keep increasing the memory. The console seems to be always outputting comments related to InternationalCode, Locations, Dates. I am sure if this is correct or part of some recursion loop.

    Perhaps, I am going about the process incorrectly, followed the YouTube videos, and if the proper procedures could be posted that might be beneficial.

    We would like to try CAM with NIEM 3.0 and any advice we could get would be greatly appreciated.

    Thanks!
    Dave

     
  • drrwebber

    drrwebber - 2014-05-24

    Apologies for the delay in us getting to this. As you are finding out its not an easy task - due to hidden issues in the NIEM schema themselves.

    You are correct though - starting with NIEM core is best. There's two things you need to know.

    1. edit structures.xsd - and comment out all those unnecessary attributes - that vastly (5x) reduces size of everything (e.g. metadata, ref-ids et al).

    2. NIEM core contains recursive elements (that's all those messages you are seeing in the import console, over and over and over). Its a slow laborious process to label these (since NIEM does not identify them for you). Basically you have to look at the console log - figure out the recursion, and then go label the head of the loop - with an annotation (add it in the NIEM core schema) on the type - with the simple word - recursive - e.g. ones I know already are organization and location - both reference themselves. With the recursive annotation in place - the CAM import will not loop on that component.

    If you have time to fight through that - fantastic! Otherwise - we are working on a CAM v3.2 release right now - bug fixes to some important XSLTs - and once we have that out - then NIEM 3.0 is our next priority. Estimate - it is going to be a couple more weeks till we start on it - and probably a week to get NIEM-core itself done - and then after that will publish each other domain dictionary once they are successfully loaded.

    Hope that is helpful,

    David

     
  • Cristian Lazurean

    Hi,

    We are also interested in NIEM dictionaries, especially on biometrics domain.
    Any news on this?

    Thank you.
    Cristian

     
  • tom

    tom - 2015-04-21

    Just to jump on the tail of this discussion. We're evaluating NIEM 3.1 (beta) and would like to use CAM in this process. So we need to import the 3.1 schemas.

    I'm starting with NIEM core and run into a problem pretty quickly with the first element. ActivityConveyanceAssociationType gets stuck in a loop dealing with it's parent element: AssociationType (actually the parent of nc:AssocationType, structures:AssociationType is the issue)

    It looks like a recursion problem with this being repeatedly output to the console:

    --- complexType AssociationType not matched for ns http://release.niem.gov/niem/structures/3.0/
    [info]extension with @base content: structures:AssociationType
    [info]xsd:complexType[xsd:complexContent]:AssociationType extn:structures:AssociationType; ns[]
    [info]Applying xsd:complexContent[complex]:;[]
    [info]extension (complex): structures:AssociationType - AssociationType
    [info]extension lookup base type: AssociationType
    [info]applying type for name: AssociationType

    As this is an inherited element, I'm not sure how to go about marking it as recursive - commenting out the extension allows the parsing to continue, so I figure there isn't an issue with importing that structures file.

    Any pointers in figuring this out? I've followed the instructions above in commenting out the attributes in the structures.xsd file.

    Any help, greatly appreciated.

    tom

     
  • drrwebber

    drrwebber - 2015-04-21

    Tom,

    I will download the 3.1 package and take a look too. Not sure what is going on there with AssociationType - usually CAM can spot automatically those direct recursive references.

    To manually label a schema component that is causing recursion and looping - you need to add:

    <xsd:annotation><xsd:documentation>recursive</xsd:documentation></xsd:annotation>
    

    to it. Once that is labelled - processing should only reference it once.

     
  • drrwebber

    drrwebber - 2015-04-21

    Tom, update on this. I just tried the niem-core.xsd with CAM 3.2 - and it crashes.

    I have CAM 3.2.1 with several important fixes for xsd extensions - and it is working. I need to focus on getting that release out the door here - we're in the middle of regression testing right now.

    Make no mistake - this NIEM 3.0/3.1 is a big challenge. I'm already seeing major raft of recursive references around Organization and Person. CAM is spotting the direct recursion - its the indirect that has to be manually labelled.

    It's deja vu sadly - as NIEM 2.0 was similar. Unfortunately NIEM 3.0 is very different - so the patches that worked to make NIEM 2.0 usable are not simply transferable.

    OK - bottom line - let's get CAM v3.2.1 shipped - then we can focus on NIEM 3.0 importing next week...

    I think if we can crack the recursive references between Person, Organization and Location it will go a long way to making things work - that was the experience the last time. Sadly there is no easy way to catch these with simple search.

    Thanks, David

     
  • tom

    tom - 2015-04-24

    Thanks David,

    I appreciate the effort.

    tom

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks