#18 Subscription-based Property Management

[Anonymous]

This is meant to turn the proposal mentioned in #1847251 into a separate discussion; in order to specifically discuss possible approaches to implement subscription-based property management for multiplayer purposes.

"However, this would also require a slightly more-sophisticated approach to
making aircraft-related properties available via network, preferably on a
subscription-basis, where multiplayer servers simply act as data
dispatchers without any formal high-level concept of the type of
aircraft/data they're distributing, while connecting clients would publish
their data on the server, and subscribe to certain data from other
clients.

This would make it possible for multiplayer servers to dispatch data in an
aircraft-agnostic (possibly even object-agnostic fashion)."

Required Features:
- allow aircraft to publish their data
--requires:
-- property name
-- property type

- allow aircraft to un/subscribe to server-side data
-- requires:
-- property name
-- property type

(to be continued)

[Nobody/Anonymous]

Logged In: NO

conceptually there are currently at least the following 5 classes of data being propagated to the server:

- aircraft position (i.e. lat/lon & altitude)
- aircraft orientation
- aircraft speeds
- aircraft model animations (exterior-flaps,slats, gear etc)
- interior/cockpit animations (i.e. panel animations)

However, not all of these may be required by all clients, some clients may only require access to certain values.
So, differentiating between such cases where all/most data is needed, and where only certain data is needed, may
be crucial to address performance issues and effiency.

For example, an ATC radar client will normally only require positional information, i.e. information that may be normally obtained from a radar dish, and possibly some additional information such as transponder settings (squawk code), likewise aircraft beyond a certain distance, will usually require less detailed information about a model and its applicable animations than those that are comparatively close.

[Nobody/Anonymous]

Logged In: NO

subscribing and unsubscribing to certain data should probably be rule/condition-based, so that client-side settings may determine whether or not the client will subscribe to certain classes of information of aircraft.

For example, a client realizing that another client's aircraft is going to be within (visible) proximity in a couple of seconds, should probably start subscribing to animations of the exterior model, while clients realizing that an aircraft is leaving the directly visible area soon, should unsubscribe from irrelevant animations that would then no longer be needed.

Likewise, there could be a configurable default range for aircraft to whose data clients will subscribe automatically, so that unnecessary traffic is excluded from subscriptions [optionally]. Each client would then determine based on locally set values which subscriptions they want to issue, maintain and remove.

This would also handle those cases properly where aircraft/clients are unnecessarily informed and updated about other aircraft hundreds of miles away, even though that data may realistically not become relevant anytime soon (i.e. think sail plane which is informed of very far away traffic)

[Nobody/Anonymous]

Logged In: NO

update intervals should probably also be configurable, this applies to both: publishing data and subscribing to data.

So that clients publishing data may do so at a (via client-side) configurable interval, depending on relevance and mutability of a property-while clients issuing subscriptions to a server may do so requesting a configurable interval that is to be used to send updates, to avoid comparatively irrelevant data to be sent at unnecessarily high intervals.

This is another scenario where depending on the multiplayer traffic situation, data requirements may vary and thus update intervals should be reconfigurable at runtime, so that a client may dynamically update a subscription (i.e. based on a client-specific subscription ID) to decrease/increase [change] the interval data is sent out at.

Alternatively, it is possible that there would be also advantages if certain data could get only sent based on conditions or rules, i.e. based on other client-side state, or possibly only issue a subscription that says "subscribe to property and update if value is changed".

Later on, one might want to combine the latter possibility with a configurable range or tolerance, so that changes must be of certain significance before they're being reflected by being broadcast to the multiplayer server

[Nobody/Anonymous]

Logged In: NO

furthermore, clients may benefit from a possibility to sort of "pre-package" subscription "lists" [at the server-side] for different data requirement scenarios, so that they may dynamically change their subscriptions by activating another list of subscriptions instead of issuing single re-subscription instructions one after another.

For example:
-- List 1 (Position)
--- /position/latitude-deg
--- /position/longitude-deg

-- List 2 (Speeds)
--- /aircraft/feet-per-minute
--- /aircraft/rollrate

-- List 3 (exterior animations)
--- ....

This could take place at initialization time, i.e. during handshake, so that changing subscriptions later on would merely come down to issuing a "switch active subscription list (based on server-side storage)".
In order to allow arbitrary combinations of lists to be active at the same time, each subscription list could be assigned numbers so that bitmasks can be used to represent each valid state.

Such pre-packaged subscription lists could in fact be shared by all clients using the same type of aircraft, so that they would be applicable to all these clients.

And instead of having each aircraft change subscriptions one by one, they would merely enable new lists for their client context, which can be done with just a couple of bytes, compared to a significant higher overhead that would be the case otherwise.

So, using such server-side lists, un/subscription would be done rather transparently behind the scenes rather than being explicitly conducted each time data requirements change.

Ideally, aircraft authors and maintainers would use a separate XML file to describe which of their aircraft's properties need to be exposed (position, orientation, speeds, exterior animations, interior animations), which could then automatically set up the corresponding "publish" routines.

[Nobody/Anonymous]

Logged In: NO

Also, there's a fixed minimum number of property names that are likely to be required by all or most vehicles rather often (i.e. position & model orientation), to save network bandwidth these could be expressed via hashes or CRCs or unique short identifiers, so that their full property names don't always need to be provided but could rather be represented via such short IDs, i.e. by having the client retrieve a "common property ID" or "CPID" from the server for a given property name such as "position/latitude-deg" so that the server in turn may reply with a special abbreviated CPID for such properties, avoiding unnecessary traffic by caching such IDs for commonly used property names, so that subscriptions may make use of such server-side IDs to save network bandwidth.

[Nobody/Anonymous]

Logged In: NO

Using a single byte, 2**8 distinct "CPIDs" could be expressed, so that in fact most current -frequently used- FlightGear aircraft and their properties could be assigned their own unique ID, type information could be either explicit by use of special instructions for subscribing to different types (int,float,double...) of properties or alternatively implicitly looked up on the server-side for each property name.

[Nobody/Anonymous]

Logged In: NO

also, there are some possible benefits if different classes of properties/state are maintained separately, so that less important/low priority client updates do not necessarily impact updates of higher priority data, i.e. assuming that one single binary is used and only forked in the shell to represent different property types (e.g. position, orientation, animations), updates wouldn't necessarily affect each other's performance.

So, a server process could automatically fork itself a couple of times for each distinct type of property to represent, using possibly increasing portnumbers (5010-5015), so that clients may implicitly -by convention- try to connect to mpserver port +x, where each 'x' might be used for a different purpose:

mpserver01:5010 - main server (positions)
mpserver01:5011 - server for important aircraft speeds
mpserver01:5012 - orientation server - aircraft orientation
mpserver01:5013 - exterior animations server
...

conceptually all servers would be identical, they would only differ in how they're used, i.e. what sort of data is being maintained via these servers, while clients would use a fixed offset to the port number for publish/subscribe requests.

[Nobody/Anonymous]

Logged In: NO

Related FGDEVEL discussion (search for "publish"/"subscribe"):
http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg13388.html

[Nobody/Anonymous]

Logged In: NO

Also please note that for maximum efficiency it might be good to keep in mind that the subscriber<->provider relationship is not merely restricted to clients subscribing to/requesting data on the server from other clients, but rather that there's also a possible benefit in recognizing that even clients PROVIDING (publishing) data may possibly do so on a subscription-basis to the server, i.e. depending on the virtual traffic situation, a connected client may not have to publish all of its properties because all traffic is relatively far away, and thus will mainly require position information to be provided, but none of the other info. Thus, providing that information unnecessarily -while straight forward from a design POV- may not be the most efficient thing to do, i.e. servers may also issue subscription requests to connected clients to determine for which data they have valid subscriptions from other clients, and thus need the data in the first place.

However, please realize that this should probably not become an immediate design goal because of the complexity involved.
In the long run it might however be interesting to take this into account for future protocol modifications.

Even if subscription based property management is for starters restricted to express a one way relationship between clients subscribing to data on the server-side from other clients, this would still be a significant improvement in so many ways.

[Nobody/Anonymous]

Logged In: NO

Initially, we will mainly need support for the following additional message types to provide support for basic subscribe/publish mechanisms:

"PUB" Message:
MessageSize | PROTO_VERSION | PUB: PROPERTY_TYPE, PROPERTY_IDENTIFIER, PROPERTY_VALUE
This message would be sent by all clients to the serer to publish data on the server for other clients
Future versions might want to make publishing certain values conditional, so that they are i.e. only transmitted if a condition is
true, e.g. if a property has changed on the client side.

"SUB" Message (to subscribe to server-side properties):
MessageSize | PROTO_VERSION | SUB: PROPERTY_TYPE, PROPERTY_IDENTIFIER, PROPERTY_VALUE | SUB:CONDITION, UPDATE_INTERVAL

"ESUB" Message (to edit/update subscription condition/interval)
MessageSize | PROTO_VERSION | "ESUB:" SUBSCRIPTION_ID | SUB:CONDITION,UPDATE_INTERVAL

"DSUB" Message (to disable a subscription temporarily, i.e. maintain subscription ID for later reuse, but don't receive data for now)
MessageSize | PROTO_VERSION | "DSUB:" SUBSCRIPTION_ID

"RSUB" Message (resubscribe, activate previously stored subscription using subscription ID)
MessageSize | PROTO_VERSION | "RSUB:" SUBSCRIPTION_ID

"UNSUB" Message (to unsubscribe from data completely, i.e. remove subscription ID):
MessageSize | PROTO_VERSION | SUBSCRIPTION_ID

Interestingly, subscription IDs could be shared for all identical properties across multiplayer session, so that the expected overhead is probably comparatively insignificant.

Also, if the previously mentioned idea of supporting pre-created subscription "packages" on the server-side is to be pursued, this would merely require the aforementioned messages to support an optional argument detailing which list they belong to, as well as simple messages to activate/change lists on the server, and some minor switch-like messages to allow management of such lists.

For example, a SUBscription message specific to a certain subscription list, may simply provide a final optional argument in the message to indicate to which server-side list the subscription is to be assigned.

Some minor messages may be required to allow clients to:
- request creation of new server-side lists
- check if a subscription list is available or not

Identical aircraft would then be able to share subscription lists, so that these could be defined in aircraft-specific XML files that would automatically be converted into transmissions that will update the corresponding server-side lists.

Such an approach would be extremely like to significantly reduce network traffic/usage due to the fact that clients would merely in the beginning have to request a certain subscription list, while later on not individual properties would be requested but rather certain subscription lists would be enabled/disabled using extremely short messages.

[Nobody/Anonymous]

Logged In: NO

NB:Interestingly, subscription-based property propagation would eventually even facilitate #1866527, so that it would for example become possible to run a FDM on the server side quite easily.

[Nobody/Anonymous]

Logged In: NO

sending type information would seem redundant, property types should normally not change at runtime.
Thus, it should be sufficient to maintain a server-side table of properties including their native types, so that at least 255 distinct properties can be expressed via one single byte, this should suffice to account for all/most aircraft and other vehicles, in particular if property usage is refactored and differently named/typed properties are redefined to share common characteristics.
So, the overall goal should be that all aircraft available in the base package use and expose as few properties as possible and hare as many properties (names & types) as possible.

So, setting a unique property defined via a corresponding "property id" would automatically take care of type information and handle proper data type usage implicitly, based on a server-side lookup table for all supported properties. Therefore, we don't need to provide type information for properties and would also not need to use separate commands/messages to account for different property types.

Basically, this means that for a comparatively small cost (i.e. maintaining a server-side table of property name/type mappings) would help tremendously to simplify the protocol rather significantly.

[Nobody/Anonymous]

Logged In: NO

but there still needs to be a refined definition of how exactly configurable subscription CONDITIONS are supposed to work, i.e. as discussed on the multiplayer webpage at: http://fgms.sourceforge.net/protocol.php

"Some properties get only sent once, during the initialisation phase. Those properties do not change during a lifetime of a session, like callsign, used aitcraft etc. Other properties get sent in every packet, like position, speed etc. And others get only sent sometimes, therefor reliably, like changes to flaps, gear etc."

So, subscriptions may depend on runtime conditions such as:
- a property has been updated (i.e. writes to a property)
- a property has been changed (i.e. not only updated, but in fact set to a different value than before)
- a property is set to a certain value (i.e. true/false)
- a property value is within/outside of a certain configurable tolerance level or range

Also, properties may possibly refer to data from a clients local tree, or from the property tree of a remote client (i.e. other traffic)

[Nobody/Anonymous]

Logged In: NO

to save bandwidth, there could be a property ID negotiation cycle/routine that will by design reply with a unique identifier for each property sent to the server, so that connecting clients could see if their property is already supported by a server-side ID (i.e. by sending a hash of the full property name to the server), where the server could compare the hash against hashes of properties supported by the server, if found the unique property ID byte is replied to the client, so that future transmissions may reuse this data instead of using full property paths.

Unsupported or special properties could thus be provided with temporary, dynamically-created unique property IDs.

The less flexible, but more straight forward solution would be to work with a static list of property name/ID pairs.

[Nobody/Anonymous]

Logged In: NO

Well, in its current form fgms is basically already using "property IDs" in order to avoid sending property strings around every time an update is sent. See the following two files for details:

http://cvs.flightgear.org/cgi-bin/viewvc/viewvc.cgi/source/src/MultiPlayer/multiplaymgr.hxx?revision=1.5&view=markup
http://cvs.flightgear.org/cgi-bin/viewvc/viewvc.cgi/source/src/MultiPlayer/multiplaymgr.cxx?revision=1.20&view=markup

This is where you can find (in the header):

struct IdPropertyList {
unsigned id;
const char* name;
SGPropertyNode::Type type;
};

So, right now property "IDs" are expressed using "unsigned int", which boils down to 4 bytes per ID/property.

As you can see in multiplaymgr.cxx, these properties and their corresponding IDs are currently defined in
a rather static fashion, so all aircraft that are supposed to be usable via multiplayer, need to make use
of only these properties. Additional new properties need to be reflected first in the code and be recompiled.

While 4 bytes may be considered wasteful from our current perspective (not yet really exploiting the full 32 bits),
using merely a single byte to express this information (even though we are not yet really using many separate
properties right now) may not be suitable for future more complex modifications, and may unnecessarily limit
the future of the mplayer facility.

On the other hand, it might make sense to consider differentiating between "core" properties (that is those
properties that are basically "standard" and high-priority among most aircraft and vehicles: positions, accelerations)
and those that are "custom" properties, defined on an aircraft-specific basis and which may not need frequent updates
by default.

The former could probably rather easily and safely expressed as single bytes (i.e. define a maximum of 255 distinct
standard/core multiplayer properties by design), while the latter could still resort to using 4 bytes for custom stuff.

This would also reflect increased priority (i.e. reduced latency/overhead due to smaller message size) for core/standard properties, compared to aircraft-specific custom properties that will not need to be updated that often.

So, any standard property (i.e. shared among most aircraft) requiring high priority updates a couple of times a second,
could be an ideal candidate to be expressed as bytes. This is also where the message size reduction of 3/4 will be
most notable (i.e. theoretically 3 bytes less per high-priority transmission).

[Nobody/Anonymous]

Logged In: NO

The most flexible and powerful approach to maintaining server-side state would probably involve equipping each server with its own instance of a property tree, so that the server would end up being a property tree server supporting publish/subscribe messages which may include optional arguments to specify requested update intervals and update conditions, where all transmitted state would end up being put into the server's property tree namespace.

The server would end up being a state variable server supporting dynamically configurable publish/subscribe methods in order to store, maintain and propagate state selectively and on a conditional basis.

While maintaining copies of all relevant property names on the server side is comparatively expensive, the previously discussed hash approach where only a hash of a full property name would be kept to identify properties and assigning them to unique byte-sized property IDs might be sufficient but less intuitive in the long run, particularly when it comes to debugging and troubleshooting servers.

So, maybe this should only be kept in mind as a possible future addition (i.e. for optimization purposes)

[Nobody/Anonymous]

Logged In: NO

To determine which functionality would eventually need to be supported by a publish/subscribe mechanism, it might be useful to evaluate the following archived mailing list discussions about ideas contributed by the fgfs community to come up with low level requirements to implement the more important stuff:

http://www.mail-archive.com/flightgear-devel@flightgear.org/msg18090.html
http://www.mail-archive.com/flightgear-devel@flightgear.org/msg18143.html

[Nobody/Anonymous]

Logged In: NO

So, a direct subscription request in the simplest case is just defined as:

"a message sent to the server indicating a property-id whose property value updates
shall be sent back to the client at a configurable interval"

more complicated versions of a subscription may involve:
- conditions to enable/disable subscriptions on the server-side based on server state
- conditions to adjust the update interval based on server-side state

[Nobody/Anonymous]

Logged In: NO

please do NOT start maintaining enhanced property-information on the server-side: one of the most powerful things about the current design is its simpleness and the fact that the server lacks any sort of data awareness, i.e. the server doesn't really possess any knowledge about what sort of variables it is maintaining and propagating.

And this is incredibly powerful, this could easily turn out to be the basis to implement a generic server architecture that can be used for arbitrary purposes (as discussed in some of the other RFEs here).

[Nobody/Anonymous]

Logged In: NO

> While maintaining copies of all relevant property names on the server side is comparatively expensive

How so?
We are talking of maybe 150-200 different properties at most - even if each is about 128 bytes in length, so about 25 kbytes in total- that should not really be much of a problem. Also, properties would then of course be referenced/pointed to, so that's really negligible .

While it would of course be inefficient to keep sending complete property names around, there is hardly any need to also maintain them on the server-side using their IDs, assuming that IDs could mismatch among clients, using hashes to assign each property ID the matching full name (as it could be provided using an XML file) should be fairly simple and would have no impact whatsoever on the traffic used.

[Nobody/Anonymous]

Logged In: NO

hashes may, depending on the underlying algorithm, also require several bytes of data to transmit, so the most promising approach would probably indeed be to use such hashes only initially during the negotiation phase of property IDs, for example like that:

- fgfs client wants to transmit a property named "/sim/foo/my-value" of type INT
-- property name and type are combined to one string to preserve type information: "/sim/foo/my-value:INT"
-- this new string is used to create a hash, i.e. using MD5: e28db21811132a7f99ac65c17109d376
-- this hash is used to identify the property and its type
- property ID negotiation starts:
-- transmit the client-side generated hash to the server and request a unique property-ID (i.e. to be represented using a byte, only)
-- the server checks locally a vector of hashes:
-- if it doesn't find the hash, the hash is put into the vector and a new unsigned char integer is created for this hash and stored in the vector
-- if the hash is found in the vector, its previously created unique ID is retrieved
-- the server replies back to the client that the transmitted hash (i.e. identified via message number) corresponds to property-ID number "x"

now from that point on, the client can store this information for this property, possibly even as a dynamically created runtime attribute for the corresponding property, i.e. dynamic-unique-multiplayer-id="1", so that future updates may refer to the negotiated ID directly.

This is pretty flexible and powerful in that the server doesn't need to maintain a server-side list of property names or types, rather the server would be completely type-agnostic and would merely map hashes to unique server-side IDs, so that it can be ensured that all clients are referring to the same property name & type later on.

This wouldn't even require server-side XML code, but if someone is interested in having an ASCII representation of each property maintained on the server-side it would only take 5 minutes to add a list of properties, so that their hashes can be calculated, and thus client hashes could be replaced with ASCII names for debugging purposes

[Nobody/Anonymous]

Logged In: NO

While 255 is probably a pretty safe number because it is unlikely that fgfs aircraft are going to send out (publish) more than 255 distinct properties anytime soon, it might be more future-safe to simply use a short, i.e. 2**16 for those property ID by default.

Alternatively, one could have the server also support property-ID type negotiation, so that it will use the smallest type possible all the time, i.e. once there are more than 255 distinct properties published on a server, automatically switch to the next logical representation, i.e. "unsigned short" in this case.

All this could happen in the same property-ID negotiation message, so that the server will automtically respond with a message indicating the size of property-IDs

Also, it might very well make sense to provide support for server-side XML, i.e. to cache the created hash/id map so that these calculations can optionally be stored and reloaded.

Whatever hashing mechanism is used, this should preferably be made very explicit during the negotiation phase (i.e. using a flag such as NEGOTIATION_METHOD_MD5, so that code updates would be easier to do.

Apart from that, it has been mentioned before but it probably makes sense to emphasize this again: frequently updated properties should be represented using the smallest possible property IDs (type size), while rarely updated properties could easily use slightly larger property IDs.

This was previously called "core" and "custom" properties, however it is important to note that some core properties may, depending on aircraft, also be registered to be updated comparatively rarely. While some custom properties may require high update rates. Thus, it might make sense to consider differentiating between subscription requests for "high rate" and "low rate" updates, i.e. by supporting a flag that explicitly sets this.

Alternatively, one could consider a very simple algorithm to automatically determine the most appropriate type size for a property ID on the server-side:

- look up number of clients which subscribed to this property
- look up all requested update intervals (which may vary among clients)
- calculate average update rate in hertz for all clients (x/y=z)
- compare this value z with the result of doing the same calculation for a newly registered property

Something like this could even be done automatically at runtime every couple of minutes to optimize data sizes and reduce network traffic, so that the server always ensures that the smallest possible types are used for frequently updated data.
- calculate

[Nobody/Anonymous]

Logged In: NO

some more thoughts:
- some clients may not only want to publish local (client-specific) state, but possibly also global state (so, there could be a way for clients to specify if they want to publish LOCAL or GLOBAL)
- only authorized clients may be allowed to publish GLOBAL data (because it might affect more than one client)

- clients might not only be interested in state of other clients, but possibly also server-specific state, so that servers can publish information, too

[Nobody/Anonymous]

Logged In: NO

using the fgfs property tree also as server-side data structure would also mean that it would be quite easily possible to provide persistence support for hosted sessions, so that users may pause their flight and quit the simulator and be able to resume the flight later on.

Basically, this would only require a way to tell the client the server-side ID that was used to reference the location in the property tree where all state was put (maybe with a flag set to avoid deletion after simulator timeout).

So, once a client sends a "resume" request to the server using the client identifier, it would look up the ID in its own property tree and transmit all relevant state back to the client, so that the client-state can be set from this state (i.e. contrary to standard behavior where server-state is initialized with client-state)

[Nobody/Anonymous]

Logged In: NO

sorry, of course that would've been the pointer only, hash table access is illustrated in the fgGet* helper functions