[Registry] Elektra 0.8.11 Release

[Markus R. <el...@ma...>]

Hello everyone,

We gladly announce Elektra 0.8.11 is released!

Elektra provides a universal and secure framework to store configuration 
parameters in a global, hierarchical key database. The core is a small library 
implemented in C. The plugin-based framework fulfills many configuration-related 
tasks to avoid any unnecessary code duplication across applications while it 
still allows the core to stay without any external dependency. Elektra 
abstracts from cross-platform-related issues with an consistent API, and 
allows applications to be aware of other applications' configurations, 
leveraging easy application integration.

See the website for more information http://www.libelektra.org

The Release notes, as they follow in this mail can also be found as html: 
http://doc.libelektra.org/news/7d4647d4-4131-411e-9c2a-2aca39446e18.html




# Release Notes

From the beginning of the Elektra Initiative, Elektra aimed at avoiding
hard-coded information in the application and to make the application's
configuration more transparent. While avoiding any pathes to files was
reality from the first released Elektra version, now also hard-coding
default values, fallback mechanisms and even Elektra's pathes to keys
can be avoided.

How does that work?

Elektra 0.8.11 introduces a so called specification for the
application's configuration. It is located below its own namespace
`spec` (next to user and system).

Once the base path is known, the user can find out all Elektra
pathes used by an application, using:

    kdb ls spec/basepath

Keys in `spec` allow us to specify which keys are read by the application,
which fallback it might have and which is the default value using
meta data. The implementation of these features happened in `ksLookup`.
When cascading keys (those starting with `/`) are used following features
are now available (in the meta data of respective `spec`-keys):

- `override/#`: use these keys *in favour* of the key itself (note that
    `#` is the syntax for arrays, e.g. `#0` for the first element,
    `#_10` for the 11th and so on)
- `namespace/#`: instead of using all namespaces in the predefined order,
    one can specify which namespaces should be searched in which order
- `fallback/#`: when no key was found in any of the (specified) namespaces
    the `fallback`-keys will be searched
- `default`: this value will be used if nothing else was found

This technique does not only give you the obvious advantages, but
also provides complete transparency how a program will fetch a configuration
value. In practice that means that:

    kdb get "/sw/app/#0/promise"

will give you the *exact same value* as the application uses when it
lookups the key `promise`. Many `if`s and hardcoded values are avoided,
we simply fetch and lookup the configuration by following code:

    Key *parentKey = keyNew("/sw/app/#0", KEY_CASCADING_NAME, KEY_END);
    kdbGet(kdb, ks, parentKey);
    
    ksLookupByName(ks, "/sw/app/#0/promise", 0);

We see in that example that only Elektra pathes are hardcoded in
the application.  But those can be found out easily, completely without
looking in the source code. The technique is simple: append a
logger plugin and the KDB base path is printed to:

- stdout in the case of the plugin tracer
- syslog in the case of the plugin syslog
- journald in the case of the plugin journald

What we do not see in the program above are the default values and
fallbacks. They are only present in the so specification (namespace `spec`).
Luckily, the specification are key/value pairs, too. So we do not have
to learn something new, e.g. using the ni plugin we can specify:

    [promise]
    default=20
    fallback/#0=/somewhere/else
    namespace/#0=user

1.) When this file is mounted to `spec/sw/app/#0` we specify, that
    for the key `/sw/app/#0/promise` only the namespace `user` should be
    used.
2.) If this key was not found, but `/somewhere/else` is present, we will use
    this key instead.  The `fallback` technique is very powerful: it allows
    us to have (recursive) links between applications. In the example above,
    the application is tricked in receiving e.g. the key `user/somewhere/else`
    when `promise` was not available.
3.) The value `20` will be used as default, even if no configuration file
    is found.

Note that the fallback, override and cascading works on *key level*,
and not like most other systems have implemented, on
configuration *file level*.


## Namespaces

The specification gives the namespaces clearer semantics and
purpose. Key names starting with a namespace are connected to a
configuration source. E.g. keys starting with:

- `user` are keys from the home directory of the current user
- `system` are keys from the `/etc` directory of the current system
- `spec` are keys from the specification directory (configurable
    with KDB_DB_SPEC, typically `/usr/share/elektra/specification`)

When a key name starts with an `/` it means that it is looked up by
specification. Such a cascading key is not really present in the keyset
(except when a default value was found). They are neither received
nor stored by `kdbGet` and `kdbSet`.

Applications shall only lookup using cascading keys (starting with `/`).
If no specification is present, cascading of all namespaces is used as before.

Elektra will (always) continue to work for applications that do not have a
specification. We strongly encourage you, however, to write such a
specification, because:

- it helps the administrator to know which keys exist
- it documents the keys (including lookup behaviour and default value)
- and many more advantages to come in future releases..

For a tutorial how to actually elektrify an application and for more
background to Elektra,
[read this document]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/tutorials/application-
integration.md).

For a full list of proposed and implemented meta-data,
[read this document]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/NAMESPACES.md).


## Simplification in the merging framework

As it turned out the concept of very granular merge strategies was hard to 
understand
for users of the three-way merging framework that emerged in the last year's
GSoC. While this granularity is desirable for flexibility, we
additionally wanted something easy to use. For that reason merge
configurations were introduced. These are simply preconfigured configurations for 
a merger
that arrange required strategies for the most common merging scenarios. 
Especially
they make sure that meta merging is handled correctly.

Have a look at the changes in the example
[src/libtools/examples/merging.cpp]
(https://github.com/ElektraInitiative/libelektra/blob/master/src/libtools/examples/merging.cpp)
for an glimpse of the simplifications.

A big thanks to Felix Berlakovich!

The header files will be installed to /usr/include/elektra/merging, but they 
are
subject to be changed in the future (e.g. as they did in this release).

From the merging improvements some minor incompatibility happened in
`kdb import`. Not all merging strategies that worked in 0.8.10 work
anymore. Luckily, now its much simpler to choose the strategies.


## API

The main API kdb.h has two added lines. First a new method was added:

    ssize_t keyAddName(Key *key, const char *addName);

This method is already used heavily in many parts. Contrary to 
`keySetBaseName` and
`keyAddBaseName` it allows us to extend the path with more than one Element at 
once,
i.e. `/` are not escaped.

The other new line is the new enum value `KEY_FLAGS`.
This feature allows bindings to use any flags in keyNew without actually
building up variable argument lists.  (Thanks to Manuel Mausz)

As always, API+ABI is stable and compatible.


## Proposed

Many new functions are proposed and can be found in
[the doxygen docu](http://doc.libelektra.org/api/0.8.11/html) and in
[kdbproposal.h]
(https://github.com/ElektraInitiative/libelektra/blob/master/src/include/kdbproposal.h).

Noteworthy is the method `keyGetNamespace` which allows us to query all
namespaces. Since this release we changed every occurrence of namespaces
(except documentation) with switch-statements around `keyGetNamespace`.
This allows us to add new more namespaces more easily. (Although its
currently not planned to add further namespaces.)

Finally, a bunch of new lookup options were added, which might not be
useful for the public API (they allow us to disable the
specification-aware features mentioned in the beginning).


## Obsolete and removed concepts

### umount

The concept that backends have a name (other than their mountpoint)
is now gone. Backends will simply be named with their escaped mountpath
below system/elektra/mountpoints without any confusing additional name.

Unmounting still works with the mountpath.

Removing this concept makes Elektra easier to understand and it also
removes some bugs. E.g. having mountpoints which do not differ except
having a `_` instead of a `/` would have caused problems with the
automatic name generation of Elektra 0.8.10.

Old mountpoints need to be removed with their 0.8.10 name
(`_` instead of `/`).

### directory keys

Additionally, the so called directory keys were also removed.
Elektra was and still is completely key/value based. The `/` separator
is only used for mountpoints.

### fstab

The plugin fstab is also improved: Slashes in mountpoints are
escaped properly with the internal escaping engine of keyAddBaseName()
(i.e. without any problematic `/` replacements).

### dirname

getDirName() was removed from C++, gi-lua, gi-python2, gi-python3,
swig-lua, swig-python2 and swig-python3. It was never present in C and
did not fit well with keyBaseName() (which returns an unescaped name,
which is not possible for the dirname). (Thanks to Manuel Mausz)

### invalid parent names

While empty (=invalid) names are still accepted as parentName to `kdbGet`
and `kdbSet` for compatibility reasons, but the parentKey

    Key *parentKey = keyNew("/", KEY_END);

should be used instead (if you want to get or store everything).
They have identical behaviour, except that
invalid names (that cannot be distinguished from empty names) will
produce a warning. In the next major version invalid parentNames
will produce an error.


## KDB Behaviour

It is now enforced that before a kdbSet() on a specific path a kdbGet()
on that path needs to be done. This was always documented that way and
is the only way to correctly detect conflicts, updates and missing
configuration files. Error #107 will be reported on violations.

Additionally, the handling with missing files was improved. Empty
keysets for a mountpoint now will remove a file. Such an empty file
is always up-to-date. Removing files has the same atomicity guarantees
as other operations.

The concurrency behaviour is at a very high level: as expected many processes
with many threads can each concurrently write to the key database,
without any inconsistent states: This is noted here because Elektra
works on standard configuration files without any guarding processes.

Filesystem problems, e.g. permission, now always lead to the same errors
(#9, #75, #109, #110), regardless of the storage plugin.


## Qt-Gui 0.0.6

Raffael Pancheri was very busy and did a lot of stabilizing work:

- Added markdown converter functionality for plugin documentation
- Integrated help (Whats this?)
- Added credits to other authors
- do not show storage/resolver plugins if a plugin of that kind has been 
selected
- added menu to newkey toolbar button to allow new array entries
- added option to include a configuration keyset when adding a plugin
- show included keys when creating the plugin configuration
- Added all storageplugins to namefilters
- Reimplement ErrorDialog
- Added undo/redo of all commands and correctly update the view
- modified ToolTip size
- Color animation on search results
- Refactored Buttons to accept shortcuts
- Updated Translations
- Colors are now customizeable
- Many small fixes

The gui is already used and the remaining small bugs (see github)
are going to be fixed soon. One of the highlights is undo for
nearly every action, so nothing prevents you from trying it out!

A huge thanks to Raffael Pancheri for his contributions. His thesis can
be found at [here](http://www.libelektra.org/ftp/elektra/pancheri2015gui.pdf).


## Bug fixing

- fix issues with escaped backslashes in front of slashes
- atomic commits across namespaces work again
- memleak on ReadFile error in ni plugin
- `kdb getmeta` reports errorcode if key, but no meta was found
- `ksLookup` now will also work if a key of the keyset is used as
    search-key (aliasing problem fixed by dup() on namelock)
- resolver regex does not match to wrongly written plugins
- jna plugin is now named libelektra-0.8.11.jar, with proper symlink to 
current version, for every CMake version
- fix bashism ($RANDOM)
- new keys are correctly renamed, fixes OpenICC (thanks to Felix Berlakovich)
- comments in host keys are correctly restored (thanks to Felix Berlakovich)
- output stream in type checking is no longer locale dependent (thanks to 
Manuel Mausz)
- cmake uninstall works again
- simplify CMAKE_DL_LIBS (thanks to Manuel Mausz)

## Further gems

- Examples were improved, added (e.g. cascading, namespace) and included in
  [Doxygen docu](http://doc.libelektra.org/api/0.8.11/html).
- [METADATA specification]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/METADATA.ini)
    was nearly completely rewritten (thanks to Felix Berlakovich)
- benchmarks were greatly enhanced (runtime+heap profiling),
    and some important performance improvements were done
- All plugins now use the cmake function `add_plugin`
    (thanks to Ian Donnelly for most of the work)
- data directory (keysets as C-files) is now shared between different
    kinds of test suites.
- many more tests were added, e.g. distribution tests, KDB API tests;
  and allocation tests were readded
- now all kdb commands accept cascading keys.
- More compiler warning-flags are added and many warnings are fixed
- cleanup of old unused `keyName` methods
- The key `system/elektra/mountpoints` itself was always created and a
    left-over on a freshly installed system after the unit tests run the
    first time. The physical presence of the key is now irrelevant and
    it won't be created automatically.
- Bash completion was greatly improved (thanks to Manuel Mausz)
- Configure scripts were refactored and are now much shorter (thanks to Manuel 
Mausz)
- New Debian build agents were added that are magnitutes faster than the old 
ones (a big thanks to Manuel Mausz)
- Many KDB tests, written in C, lua and python were added (thanks to Manuel 
Mausz)
- SWIG3 is preferred when available
- add the plugin counter that counts how often the methods of a plugin are 
called
- `kdb list-tools` is now advertised in `kdb --help`
- Mac OS X support was greatly improved, thanks to Peter Nirschl
  and Kai-Uwe Behrmann. The feature rich resolver, now also works
  for Mac OS X.  wresolver is now only needed for mingw.
- Elektra still compiles with gcc (also mingw variants), icc and clang.


## Further Notes

With 471 files changed, 27978 insertions(+), 11512 deletions(-) this
release is huge. With 773 commits over four month much more changes
happened which did not find their place in these release notes, even
though the notes are much less detailed than usual.

Thanks for all contributions that are not enlisted here!

For any questions and comments, please contact the
[Mailing List](https://lists.sourceforge.net/lists/listinfo/registry-list)
or el...@ma....


## Get It!

You can download the release from
[here](http://www.markus-raab.org/ftp/elektra/releases/elektra-0.8.11.tar.gz)

- name: elektra-0.8.11.tar.gz
- size: 2022129
- md5sum: c53a8151aab760851842d745e904a4f8
- sha1: d7929d17d1a6529089d156f1910d87f678b84998
- sha256: c20fefcfba62cc906228f9b55d1f411ef8f884ff9d75774a0dd4f8eb8f5b48f6



This release tarball now is also available
[signed by me using gpg](http://www.markus-
raab.org/ftp/elektra/releases/elektra-0.8.11.tar.gz.gpg)

already built API-Docu can be found [here]
(http://doc.libelektra.org/api/0.8.11/html/)


## Stay tuned! ##

Subscribe to the
[new RSS feed](http://doc.libelektra.org/news/feed.rss)
to always get the release notifications.

[Permalink to this NEWS entry]
(http://doc.libelektra.org/news/7d4647d4-4131-411e-9c2a-2aca39446e18.html)

For more information, see http://www.libelektra.org

Best regards,
Markus

-- 
Markus Raab            http://www.libelektra.org
Technische Universität Wien       el...@ma...
Institut für Computersprachen           Phone: (+431) 58801/185185
Argentinierstr. 8, 1040 Wien, Austria     FAX: (+431) 58801/18598

DVR 0005886