Using exip in schema mode

[Rumen Kyusakov]

Document instructions of how to use schema-mode, report errors, bugs etc.

[Rumen Kyusakov]

Instructions on using exipe and exipd examples (and hence exip in general) in schema mode.

The -schema parameter accepts space-separated list of schema documents encoded in EXI with Preserve.prefixes. The first schema is the main one and the rest are schemas that are referenced from the main one through the \<xs:import> statement.

1. Encoding with exipe. Use the following command line command (assuming all the schema files are in the local directory):
   ./exipe -schema exipe-test-xsd.exi exipe-test-nested-xsd.exi exipe-test-types-xsd.exi exipe-output.exi

2. Decoding with exipd. Use the following command line command to decode the exipe-output.exi from step 1 or any other EXI stream:
   ./exipd -schema exipe-test-xsd.exi exipe-test-nested-xsd.exi exipe-test-types-xsd.exi exipe-output.exi

[Anonymous]

Hi,
I extended these examples with larger schemas.
Now it seems that exip can encode / decode
its own messages succesfully. But if I try to
decode exip encoded message with ExiProcessor
it either fails with exception or the result
is totally wrong. Also if I encode the valid
xml messge to exi with ExiProcessor then
exip cannot decode the exi, but ExiProcessor
again can decode it correctly.

I think there are three options now:
1. There is a configuration mismatch. I've used
strict option both in ExiProcessor (-strict -schema)
and exip (SET_STRICT(msg_strm->header.opts.enumOpt);)
and to may understanding that should be enough.
2. There is a problem in exip grammar generation,
however, it does not give any error messages.
3. Either exip or ExiProcessor or both are doing
encoding wrongly

Have you tested encoder / decoder against other
implementation? Is there any known issues with
the grammar generation or serialization which
could cause interoperability problem?

Regards,
Raino

[Rumen Kyusakov]

Hi Raino,

You are right, all the 3 options are possible. There are no known interoperability issues, but I am sure there are unknown ones.
I have done some testing against EXIficient (the core of ExiProcessor) but they are not exhaustive.
Some hints:
Regarding 1): Check that you set -header_options in ExiProcessor along with -schema -strict. Also you need msg_strm->header.has_options = TRUE; in EXIP.

Regarding 2): That would be tricky to trace. You can try to exclude some parts from the schema to isolate the problem.

Regarding 3): The chances are that there is some bug in exip. If you can't find the solution - compile more detailed information on the schema that you use, the options etc. and file a bug report.

Regards,
Rumen

[Anonymous]

Hi Rumen,
I have now tried the version 0.5.1 but I still experience
problems with quite simple schemas. The target is to have interoperability
with the existing java based system hence it is not enough
that exip encoder/decoder interoperate.

One issue which I noticed was that if I added a new definition to exipe-test-nested.xsd (so both testSetup and testREF are defined in xsd):

<element name="testREF">
    <complexType>
        <simpleContent>
            <extension base="string">
                <attribute name="goal" type="string"></attribute>
            </extension>
        </simpleContent>
    </complexType>
</element>

Then neither exificient nor openexi could decode exipe encoded
message correctly. Exipd on the other hand produced the correct
result. The only difference with the testSetup definition is different name.
The situation is the same even in case when there is no reference to this newly
defined element in exipe-test.xsd.

I have to use in top level schema (like 'exipe-test.xsd') elements which
are defined in different namespace (like in testSetup example) so I cannot
replace references with element with a type.

Is there a simple workaround for this problem?

Thanks
Raino

[Rumen Kyusakov]

Hi Raino,

There was a bug in version 0.5.1 and before that. You can get the latest code from the SVN with the fix for it.

I just tested: added testREF definition to exipe-test-nested.xsd as you suggested. Then convert exipe-test-nested.xsd to .exi and run the exipe on it. Then decoded the resulting exipe-test.exi with OpenEXI and it worked.

Let me know if you still have issues.

A new release with this fix and much more is coming soon.

Best regards,
Rumen

[Anonymous]

Hi Rumen,
SVN version worked correctly. I should have downloaded the code from svn in the first place.
But a new question. minInclusive/maxInclusive minExclusive/maxExclusive pairs
are not working. I got an error:
Error EXIP_INVALID_EXI_INPUT:9
when using minInclusive/maxInclusive restrictions.

Is there a bug or is the support missing?

Thanks
Raino

[Rumen Kyusakov]

Hi Raino,

minInclusive/maxInclusive are implemented but barely tested.
minExclusive/maxExclusive are not supported yet.

You can fill a bug report with a stripped version of your test case including the schema etc.

Then I can have a look and fix the issue.

Best,
Rumen

[Anonymous]

Hi Rumen,
modified exipe-test.xsd is attached.

If minInclusive / maxInclusive are within comments then there
is no problem, otherwise exipe gives
Error EXIP_INVALID_EXI_INPUT:9
at ../../examples/simpleEncoding/encodeTestEXI.c, line 115

error.

BR Raino

[Anonymous]

Hi Rumen,
some extra information.
If base type is "byte" then exipe fails.
If base type is "short" or "int" then exipe succeeds and
exipd decodes the message correctly. However, in latter
cases exificient cannot decode exi message.

br Raino

[Rumen Kyusakov]

Hi Raino,

Thanks for the feedback!
As I have tons of other things to do it will take some time before I look at it.
I filed a bug report at:
https://sourceforge.net/p/exip/bugs/12/

Best,
Rumen

[Rumen Kyusakov]

Hi Raino,

The minInclusive / maxInclusive bug is fixed now.

Best,
Rumen

[Anonymous]

Hi Rumen,
thanks for the quick correction.

But now another issue. I am trying to figure out a workaround
for substitutionGroup and abstract type because exip does not
support those types yet. I attach two versions of exipe-test
schemas to clarify the problem.

exipe-test-group.xsd and exipe-test-nested-group.xsd have
substitutionGroup definition, that is abstract type element
baseElement is replaced with baseTest element in serialized XML file.

In exipe-test.xsd exip-test-nested.xsd I have tried to make
schema definition which results an identical XML.

Now the attached exipe-test.xml can be verified against both
XML schema versions without errors.

With EXI the situation is different because elements are not
referred with full names but (to my understanding) element
indexes. But if I understood correctly EXI format specification
all elements are handled equally regardless how they
have been specified that is there is no specific mechanism for
handling substitutionGroup. All the elements (element names) are still
present in exipe-test.xsd and exipe-test-nested.xsd schemas.
However, the grammar generated from those is not interoperable
with -group.xsd schemas. That is exi message generated with
schemas exipe-test-xsd.exi and exipe-test-nested-xsd.exi
(and of course exipe-test-types-xsd.exi) cannot be decoded
correctly with grammar from exipe-test-group.xsd and
exipe-test-nested-group.xsd. Latter schemas I have used
with exificient.

Am I totally wrong when assuming that same exi grammar (or at
least interoperable grammars) could be generated from two
different schemas? Do you have any idea how to make a workaround
for substitutionGroup definition?

Attached is also modified code of encodeTestEXI.c. All files are
in zip file.

Thanks
Raino

[Rumen Kyusakov]

Hi Raino,

Interesting problem. I think most likely interoperable EXI grammars could be generated from two different schemas - one with substitutionGroup and the other without. However, you need to have quite deep understanding of the process of EXI grammar generation and even then it is quite complex and time consuming process.
There is actually a specific mechanism for handling substitutionGroups when generating EXI grammars.

I do not have a solution to your problem. My advice is do not spend time on trying to find such a workaround - it is a tricky stuff.

I do not plan to implement the substitutionGroups features any time soon so the best is to try to avoid it.

Best,
Rumen

[Anonymous]

How does one get a good understanding of the EXI encoding?

I'm decoding a relativly short EXI grammer for use in an embedded system. I'm using EXIficient to generate the EXI with a schema. Exipd is not able to decode any of these generated EXI files, however, I do have other programs able to do this.

The problem I'm running into is there many MANY instances where an element has a minOccurs=0 set. This throws off my entire understanding of the EXI format since if, for example, several occur in a row, the bit stream changes radically.

It's not clear in the EXI standard, from what I can tell, how this is precisely encoded with a schema with the options header decoded. Aggraving my problem is that there is an GPL version of this code which I have verified is entirely wrong, so I have wasted 3 weeks with that.

[Rumen Kyusakov]

Hi,

It is hard to understand your problem:

• Maybe you do not pass the appropriate command line arguments to exipd
• minOccurs=0 simply implies that there are no restrictions on the minimum occurance for an element
• the EXI specification is hard to read but self-contained. Maybe you can look at the EXI primer:

https://www.w3.org/TR/2014/WD-exi-primer-20140424/

Best regards,
Rumen

[Anonymous]

I have been going through the EXI primer.

What I now understand is that the EXI grammer changes depending on what the most common production codes are. If, for example, they are all equally likely, what would be the assigned event code for each and in what order?

I cannot find that defined.

I understand what minOccurs=0 does, what is confusing is that if my schema has two OPTIONAL binary values indicated, if the first bool is defined, and the second isn't, this encodes to 8 bits, however, if the first bool is NOT defined, and the second one is, this encodes to 9 bits. I cannot help but think this could be more efficiently defined with simply 4 bits.

In any case, I am trudging through https://www.w3.org/TR/exi-primer/#exiByExample