#332 Decide semantics of assignment/rule/reaction with no <math>

[Lucian Smith]

There are many cases in the SBML specification when a rule or an assignment overrides the previously-declared value of an element. Now that the $child\; of\; those\; elements\; is\; optional,\; we\; need\; to\; decide\; what\; happens\; when\; something\; that\; used\; to\; have\; a\; value\; is\; declared\; as\; having\; a\; new\; value,\; but\; that\; value\; is\; not\; included\; in\; the\; model.\; For\; example:$

• A parameter with a defined 'value' attribute, that also has an InitialAssignment with no $child.$
• A parameter with a defined 'value' attribute, that also has an AssignmentRule with no $child.$
• A parameter with a defined 'value' attribute and/or a define InitialAssignment, with a RateRule with no $child.$
• A species that appears as the product or reactant of a Reaction with no KineticLaw, or in a Reaction that has a KineticLaw with no $child.$
• A parameter with a defined value ('value', InitialAssignment, maybe even a RateRule, or that appears in a Reaction) which gets assigned with an EventAssignment with no $.$

I think it will be helpful if the spec explicitly calls out those situations. I think we basically have three options:
1) The value of the element at the point at which it is changed by something becomes undefined.
2) Simulation of a model in that situation may proceed as if the element with no $child\; did\; not\; exist.$
3) Simulations of such models are impossible, and an error should be given to the users.

The second and third options are both possible as an optional response of particular simulators if we go with 1), but those results would not be exchangeable. If we go with option 2), everybody must interpret the situation the same way, which makes simulations exchangeable again; if we go with 3) we send an explicit message with the spec that nobody should even try to simulate those models.

It is important to keep in mind that simulation is not the only use of SBML, but it remains the case that simulators will end up with these models, and need some explicit guidance about what to do.

It might be instructive to find out what the 'de-facto standard' is for what simulators do with models that have no KineticLaw, since those models already exist. If everyone does their own thing, we probably should go with 1. If everyone rejects the model outright, maybe we should go with 3. And if everybody proceeds as if the reaction wasn't there, we could go with 2. (Another option would be to go with 1, and have 2 or 3 as 'best practice'.)

[Chris Myers]

I don’t think this is really any different than a parameter with no value. Namely, the model is incomplete, so a simulator should notify the user of this fact. If it tries to carry on, odds are that it will be silently hiding a problem with the model. I cannot think of a simulation use case in which one would want to simulate an incomplete model. As you mention, SBML is for more than just simulation, and I do understand the value of incomplete models for these other purposes, but not for simulation.

[Lucian Smith]

Here's a concrete example of what I'm trying to get at: is the following model underdetermined, or can it be simulated?

<?xml version="1.0" encoding="UTF-8"?>
<sbml xmlns="http://www.sbml.org/sbml/level3/version2/core" level="3" version="2">
  <model>
    <listOfParameters>
      <parameter id="a" value="2" constant="true"/>
    </listOfParameters>
    <listOfInitialAssignments>
      <initialAssignment symbol="a"/>
    </listOfInitialAssignments>
  </model>
</sbml>

[Andreas Dräger]

My intuition would say, solution 1) is most plausible where here "undefined" would mean the value becomes NaN. What ever happens to NaN values in the next step of the calculation is the result of the model.

[Lucian Smith]

Claiming that the value is actually NaN is actually a slightly stronger statement than stating that the value is undefined. Many SBML simulators take 'undefined' values in SBML models and fill them in with default values (while telling the user they are doing so). Stating explicitly that you should insert NaN would mean that this practice is actually incorrect, which we could say, but I think we probably shouldn't. It would be a relatively significant break with tradition, and might interfere with our other goal with L3v2: to let packages fill in undefined values for SBML elements.

[Dagmar Waltemath]

From the three choices above, I opt for (3) Simulations of such models are impossible, and an error should be given to the users. -- Simply because I do not like too many implicit assumptions.

Citing Chris: "the model is incomplete, so a simulator should notify the user of this fact. "
+1.

Even more so, I think that it may not be the role of the specification to tell people what to do with incomplete/wrong SBML models. If the above underdetermined model is a good model in the sense of SBML, that is probably all we should care about. A hint that underdetermined models may result from the current SBML spec is good, but I would not go as far as telling people what to do with it in the simulation tool.

(But I maybe wrong ;-))

[Brett Olivier]

I have a bit of a problem with "an error should be given" does running a simulation which returns NaN's consitute enough of an error to fall under option 3? If so, then I could support it.

[Nicolas Le Novère]

I would tend towards 2. The element with no child does not modify the set of equations defining the model. Effectively it is invisible. It can be useful for future developments or to carry annotations. Having not child is different from explicitely declaring a non-value I think. An absence is not a negation.

[Lucian Smith]

For the record, Nicolas's conclusion is where I started when attempting to write up what things meant in the spec. There are statements like, "An Event with no Trigger will never fire, and neither will an Event with a Trigger with no 'math' child." or "An InitialAssignment with no 'math' child does not assign any value to its 'symbol'." I did have to be careful to say that this only applied in the absence of a package supplying the missing information.

Then I started to think, "Well, hmm, is this fair? Or is an InitialAssignment with no 'math' a stronger statement; more along the lines of 'This element has an initial value, but I do not know what it is,' instead of 'I am not supplying any information about the initial value of this element'.

I really want to know what existing software does when it encounters a reaction with no kinetic law. Does anyone know off the top of their heads, or do I have to do some actual research? ;-)

[Chris Myers]

Our simulator gives an error message that the kinetic law is missing and halts.

[Lucian Smith]

I guess the L3v1 spec does say "However, missing kinetic laws preclude the application of many model analysis techniques, including simulation."

[Lucian Smith]

Looks like Copasi treats it as if the reaction rate was zero. No warnings or anything, either.

[Lucian Smith]

Same with libroadrunner: runs without complaint; assumes reaction rate of zero. (Also, I know libroadrunner will generally supply default values to everything if not supplied by the user.)

[Lucian Smith]

So, OK. Given that of the three simulators we investigated, two of them (essentially) follow option 2 ('proceed as if it didn't exist'), and one of them follows option 3 ('stop and give an error'), this colors our choices:

1) Say it's undefined; let people do what they want: this follows current behavior, which decreases exchangeability of underdefined models between tools. (i.e. a user creates a model in libroadrunner, takes it to iBioSim, which suddenly complains.) On the other hand, different user bases for the different tools may expect these different behaviors, and tool authors may wish to defend their actions.
2) Tell people to ignore underdefined constructs: this follows Copasi and libroadrunner behavior, and would encourage iBioSim to change its behavior.
3) Tell people to error out on underdefined constructs: this follows iBioSim, and encourages Copasi and libroadrunner to change their behavior.

Choosing either option 2 or 3 may improve exchangeability, but could decrease the user experience on the tools that have to change. OTOH, if we tell people to do this only for the new situations that are now newly valid in l3v2, we may prevent the sort of divergent behavior we now see. On the other other hand, we are still telling people what to do with their tools, and they may wish to provide a consistent user experience one way or the other ('try to guess what they want' vs. 'do only what I'm explicitly told'). Either approach may be valid in different contexts.

[Chris Myers]

The reason why I believe this should be an error is that missing math is more often than not an error in the model. If you go ahead and simulate, you are making it harder for the user to realize that they have forgotten to enter something. If they want the kinetic law to evaluate to zero, why not make them enter 0. This is in the spirit of “no defaults”. Assuming no law evaluates to zero, rather than an undefined value, means it has a default.

Whatever we decide, I think we should not make there be a different behavior for “no kinetic law” and “a kinetic law but no math”. This would certainly create confusion. My preference for reasons above is that they both evaluate to an “undefined value” meaning simulation should report an error to the user.

[Brett Olivier]

With PySCeS if there is a reaction without a kinetic law I show a warning on load and return a NaN during the simulation, which essentially leads to all output being NaN.

However, this behaviour is specific to reactions as it allows one to do pure structural analysis with PySCeS on models without dummy rate equations being defined. Any other missing math or values throw errors on model load and/or simulation.

However, if there were InitialAssignments on stoichiometric coefficients that had missing math, this would also raise errors. I'm not a great fan of implicit behaviours.

[Sarah Keating]

This one is a mine field ! In L3V1 SBML said a model with a missing math element is not valid. This left the field clear for simulator developers to choose the behaviour they wanted when dealing with an invalid model.

Now for L3V2 we are saying that the model is valid and it is not unrealistic to expect the specification to answer the question; what does it mean if this is missing. However, us doing that is not necessarily going to influence the simulator developers, who have already clearly made decisions about how to handle the situation. They will not necessarily change their approach because we said so!

You could also end up in the ridiculous situation where someone has a valid L3V2 model with missing math that a software refuses to simulate (because we have said that is what should happen) but the way round it would be to convert it to an invalid L3V1 model that can be simulated by certain simulators.

If we are allowing things to be missing (!) then we have to go with 'this is undefined'.

[Nicolas Le Novère]

But in the case where we have an existing parameter value and an initialAssignment with no math, we have a defined value. If we do not have a parameter value either, then the value is undefined, no matter what the decision is on the initialAssignment.

So saying "ignore a construct with a missing math in a mathematical context" does not impact maths I think. (note the "in a mathematical context". which may be a way out of the current dilemna by cutting some slack to people who do not actually want to perform simulations)

I am sympathetic with Chris' concern, but this is a tool issue. COPASI lists plenty of such situations. It warns the user and points to possible problems. But it does not stop the user from using possible problematic models if so they wish.

[Chris Myers]

However, I think a “best practice” should be described that states that when it is missing a tool should report it is missing. It is then free to make an assumption and carry forward, if it likes. But it is best that the tool is stating to the user that an assumption has been made.

This would allow tools who want to carry forward assuming 0, NaN, or simply failing to all be correct. It is just best if the tool tells the user that they are going off the reservation and user experiences with different tools may vary at this point.

[Lucian Smith]

I like Nicolas's 'mathematical context' caveat.

OK, it seems like we're coalescing around a sort of a consensus here. What if I added the following to the definition of all SBML elements with now-optional 'math' children:

"In a mathematical context, the meaning of [this SBML object] with no 'math' child is undefined. An SBML Level 3 package may supply the missing information, but if not, individual tools may choose what to do, including ignoring the construct entirely, producing an error, prompting the user for additional input, or proceeding with a default value. It is considered best practice to inform the user about what the tool is doing in these situations."

This would be slightly modified in different contexts, for example, for the 'Trigger', I would change the middle part to '...ignoring the Event entirely...' and remove the 'proceeding with a default' clause, because there's no reasonable default 'Trigger' formula I can imagine.

[Brett Olivier]

This works for me.

[Chris Myers]

Works for me too.

[Lucian Smith]

So, I went to go actually write up this change, and it's starting to rub me the wrong way again. As an example, originally, I had:

"Similarly, a \Priority with no \token{math} child leaves the priority of the event undefined. A package may define how to include extra information that would define how to apply this \Priority; otherwise, the event behaves as if it did not have a priority."

and then changed it to:

"Similarly, a \Priority with no \token{math} child leaves the priority of the event undefined. An SBML Level~3 package may supply this missing information. In a mathematical context, individual tools may choose what to do in this situation, including ignoring the \Priority entirely, producing an error, or prompting the user for additional input. It is considered best practice to inform the user about what the tool is doing in these situations."

and I can't help but think that the new version is nothing more than a time bomb for future problems and incompatibilities. In the original version, we can make a test case with expected output for an Event with a Priority that has no math. In the new version, we can create a model that nobody knows what to do with it: produce an error? Treat the Event as if it had no Priority? Ignore the Event entirely?

Similarly, we now would have the case where an Event with no Trigger would simply never fire, but an Event with a Trigger with no math can produce an error or refuse to be simulated at all.

I do understand that appeal of not wanting to assume anything, but if we already knew what to do with a model without the construct, I'm no longer sure it's a bad idea to just continue to treat the model the same way if it has a partially-defined construct.

Could we get away with just letting tool writers produce a warning if they wanted?

"Similarly, a \Priority with no \token{math} child leaves the priority of the event undefined. A package may define how to include extra information that would define how to apply this \Priority; otherwise, the event behaves as if it did not have a priority. However, a tool may produce a warning in this situation if it so chooses."

[Chris Myers]

I think you could go a little further and say a tool SHOULD produce a warning and COULD simply decide such a model is incomplete and not simulate it at all.

I really think it is problematic to allow simulator developers to silently interpret the absence of something as having meaning. I also do not like absence of trigger to mean cannot trigger, this would infer that the trigger has a default of “false”. If we really believe in no defaults, then it should be undefined whether it should trigger or not.

I think it is indeed a time bomb to not at least strongly encourage software developers to notify the user when they are leaving the space defined in the specification.

[Lucian Smith]

There are a lot of things whose absence implies meaning, even in L3v1: the absence of an initial assignment for a value means that nothing is initially assigned to that value. The absence of a Priority means that an event, if triggered simultaneously with other events, may be executed in any order. The absence of a Delay means there is no time delay between the event trigger and it being executed. The absence of an assignment rule to a variable means that variable does not get anything assigned to it. The absence of an algebraic rule means there are no additional constraints on the variables in the model.

In L3v2, we now allow the absence of a Trigger, so we have to say what that means. We don't (in the current draft) say that it means that the Trigger is always 'false'; we say that it can never transition from 'false' to 'true', and thus the event never fires. If a Trigger is present but has no 'math' child, the same thing could be true. It's a simple explanation that makes intuitive sense, and can be tested.