With the introduction of each new oval version, it appears that each definition, regardless of whether it has changed, has a new schema_version. For example, the definition "oval:org.mitre.oval:def:7169" was created in 2010, but with the release of 5.10, the schema_version has been updated to "5.10" as of today's version upgrade.
The problem is that products using the feed will be broken instantly. Until the version has been developed, tested, release and upgraded, these products will have to ignore the version. Doing so is likely to result in new and changed tests breaking, but given the alternative of rejecting all tests, changed or otherwise, this seems marginally better.
Red Hat's repository, in contrast, seems to use the schema_version to indicate compatibility. So even if a test is newly released under a new version of the schema, if it is backwards compatible with an older version, that older version is specified, preventing the above problem.
My question is: what is the recommended methodology for products using the repository feeds after a schema upgrade? Obviously, it is impossible to immediately upgrade because an entire product release cycle is needed after the upgrade, leaving tests broken or disabled for an unacceptable period. Using the archive seems infeasible since the archive doesn't allow for selecting sub-sets of tests, as far as I can tell, and would require a URL change.
Do you have a recommended process that would allow deployed product to function through an upgrade without downtime? Without one, it makes the feed of limited value for a deployed product. Is there another way compatibility should be determined?
Thank you very much,
Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have an interpreter that was working fine (5.9) up until a couple of days ago. Definitions could be downloaded from the repository by our customers and processed as expected.
However, we started getting reports that none of the OVAL content in the repository worked anymore. Even definitions that hadn't changed in over a year.
This seems like a fundamental flaw of either the schema, the repository, or both. I don't see how a commercial product can be expected to be updated and distributed to customers by the time the oval repository decides to flip the switch on a new version. This just doesn't seem reasonable.
I would really like to know what approach MITRE and the OVAL team recommend for real-world products that rely on this technology.
Ted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This is valuable feedback. I encourage you to post these same messages to the oval-discussion-list. This forum was established for help with the OVAL Interpreter project. the oval-discussion-list was established for all discussions related to the OVAL Repository and its content. You can register for the oval-discussion-list here:
The short answer is that we have followed this practice of automatically updating the version of the content served by the repository when the language is updated for many years. We are aware that this is increasingly causing a problem for tools developers that support only a specific version of OVAL and for end users that rely upon the content from the repository working in their tools. We currently have a development task to implement support for what has been called a least version principle in the oval repository. Once completed we will be able to compute the lowest required version for a set of content and then create download files setting the schema version value based upon the real content of the file. Hearing your feedback helps us prioritize this development work. re posting this to the oval-discussion-list will ensure that others are aware of the feedback and given the opportunity to also weigh in on the best solution.
Thank you for sharing,
Jon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the feedback, I will post this to the appropriate discussion list.
However, one sentence is confusing to me in your response:
"We are aware that this is increasingly causing a problem for tools developers that support only a specific version of OVAL and for end users that rely upon the content from the repository working in their tools."
I guess, I am confused as to how a piece of software that is released and installed at a customer sites can support future versions that are yet to be designed. Therefore, the best you can do as a developer is be backward compatible to the latest version (we support all 5.x versions up to the latest at the time of our release, 5.9).
This is going to be a problem for anyone developing tools that use the repository, unless they don't check versions at all and pray that the content being delivered is compatible.
Ted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You are correct, the is really not much that a tool can do to be assured that it will properly evaluate a set of definitions written for a later version of the language. A tool could try to process content from later versions of the language. If a tools was developed to support version 5.9 of oval and received a document that was written against 5.10, the tool could attempt to validate this document against the 5.9 schema. If the document validated correctly, the tool could then process the definitions with some degree of confidence. For the time being this is actually a very safe approach to handling the content from the OVAL Repository, because as you correctly noted, the content itself did not change when we released version 5.10. We simply updated the schema_version.
As an fyi, we will describe the planned capabilities to support tracking least version information on the oval-discussion-list as soon as we have finished discussing them internally here at MITRE.
Jon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I wanted to let you both know that last week we announced the initial launch of OVAL Repository downloads by schema version. This was in direct response to your requests.
With the introduction of each new oval version, it appears that each definition, regardless of whether it has changed, has a new schema_version. For example, the definition "oval:org.mitre.oval:def:7169" was created in 2010, but with the release of 5.10, the schema_version has been updated to "5.10" as of today's version upgrade.
The problem is that products using the feed will be broken instantly. Until the version has been developed, tested, release and upgraded, these products will have to ignore the version. Doing so is likely to result in new and changed tests breaking, but given the alternative of rejecting all tests, changed or otherwise, this seems marginally better.
Red Hat's repository, in contrast, seems to use the schema_version to indicate compatibility. So even if a test is newly released under a new version of the schema, if it is backwards compatible with an older version, that older version is specified, preventing the above problem.
My question is: what is the recommended methodology for products using the repository feeds after a schema upgrade? Obviously, it is impossible to immediately upgrade because an entire product release cycle is needed after the upgrade, leaving tests broken or disabled for an unacceptable period. Using the archive seems infeasible since the archive doesn't allow for selecting sub-sets of tests, as far as I can tell, and would require a URL change.
Do you have a recommended process that would allow deployed product to function through an upgrade without downtime? Without one, it makes the feed of limited value for a deployed product. Is there another way compatibility should be determined?
Thank you very much,
Daniel
We also have this issue.
We have an interpreter that was working fine (5.9) up until a couple of days ago. Definitions could be downloaded from the repository by our customers and processed as expected.
However, we started getting reports that none of the OVAL content in the repository worked anymore. Even definitions that hadn't changed in over a year.
This seems like a fundamental flaw of either the schema, the repository, or both. I don't see how a commercial product can be expected to be updated and distributed to customers by the time the oval repository decides to flip the switch on a new version. This just doesn't seem reasonable.
I would really like to know what approach MITRE and the OVAL team recommend for real-world products that rely on this technology.
Ted
Daniel & Ted,
This is valuable feedback. I encourage you to post these same messages to the oval-discussion-list. This forum was established for help with the OVAL Interpreter project. the oval-discussion-list was established for all discussions related to the OVAL Repository and its content. You can register for the oval-discussion-list here:
https://oval.mitre.org/community/registration.html
The short answer is that we have followed this practice of automatically updating the version of the content served by the repository when the language is updated for many years. We are aware that this is increasingly causing a problem for tools developers that support only a specific version of OVAL and for end users that rely upon the content from the repository working in their tools. We currently have a development task to implement support for what has been called a least version principle in the oval repository. Once completed we will be able to compute the lowest required version for a set of content and then create download files setting the schema version value based upon the real content of the file. Hearing your feedback helps us prioritize this development work. re posting this to the oval-discussion-list will ensure that others are aware of the feedback and given the opportunity to also weigh in on the best solution.
Thank you for sharing,
Jon
Jonathan-
Thanks for the feedback, I will post this to the appropriate discussion list.
However, one sentence is confusing to me in your response:
"We are aware that this is increasingly causing a problem for tools developers that support only a specific version of OVAL and for end users that rely upon the content from the repository working in their tools."
I guess, I am confused as to how a piece of software that is released and installed at a customer sites can support future versions that are yet to be designed. Therefore, the best you can do as a developer is be backward compatible to the latest version (we support all 5.x versions up to the latest at the time of our release, 5.9).
This is going to be a problem for anyone developing tools that use the repository, unless they don't check versions at all and pray that the content being delivered is compatible.
Ted
Ted,
You are correct, the is really not much that a tool can do to be assured that it will properly evaluate a set of definitions written for a later version of the language. A tool could try to process content from later versions of the language. If a tools was developed to support version 5.9 of oval and received a document that was written against 5.10, the tool could attempt to validate this document against the 5.9 schema. If the document validated correctly, the tool could then process the definitions with some degree of confidence. For the time being this is actually a very safe approach to handling the content from the OVAL Repository, because as you correctly noted, the content itself did not change when we released version 5.10. We simply updated the schema_version.
As an fyi, we will describe the planned capabilities to support tracking least version information on the oval-discussion-list as soon as we have finished discussing them internally here at MITRE.
Jon
I wanted to let you both know that last week we announced the initial launch of OVAL Repository downloads by schema version. This was in direct response to your requests.
See: https://oval.mitre.org/rep-data/index.html
and: http://making-security-measurable.1364806.n2.nabble.com/Updates-to-OVAL-Repository-Static-Downloads-tp6917181p6917181.html
Please let us know if you have any questions or comments about the new downloads.
Thanks,
Jon