opensync-devel — The place to coordinate the development of OpenSync

[Opensync-devel] [trunk] OSYNC_START_TYPE_EXTERNAL example / OSyncClient API changes

[Daniel G. <go...@b1...>]

Hi,


finally we have an example implementation for OSYNC_START_TYPE_EXTERNAL.

What this OSYNC_START_TYPE_EXTERNAL about?

This allows to write (standalone) userspace applications which act as OpenSync 
plugin. You can use this for example if you try to gather data from an 
existing application, which doesn't provides any public interface - e.g. an 
extension of an application which communicates with OpenSync.

Currently, there is need of stub plugin, which registers the plugin to the 
OpenSync environment. As Graham already suggested in a seperated thread, we'll 
very likely change it - so there is only need to write a simple configuration 
file to register external plugins.

Additionally we'll add some configuration options so OpenSync can spawn such 
external process if they haven't be launced yet. For now such applications 
have to be started manually ...

See:
http://opensync.org/browser/trunk/docs/examples/plugins/src/external_demo.c
and
http://opensync.org/changeset/5895

-------8<-------
$ ./example_external_demo ~/.config/opensync/0.40/group6/1/pluginpipe &
[1] 10841
$
[EXTERNAL-DEMO]: OSyncPlugin:0x40159d OSyncClient:0x603040
[EXTERNAL-DEMO]: Starting (blocking) OSyncClient ...
$
$ osynctool --sync external
Synchronizing group "external"
[EXTERNAL-DEMO]: initialize
data sink of member 2 of type file-sync just connected
Main sink of member 2 of type file-sync just connected
data sink of member 1 of type external-demo just connected
Main sink of member 1 of type external-demo just connected
All clients connected or error
[EXTERNAL-DEMO]: get_changes slow_sync: no
data sink of member 2 of type file-sync just sent all changes
data sink of member 1 of type external-demo just sent all changes
Main sink of member 1 of type external-demo just sent all changes
Main sink of member 2 of type file-sync just sent all changes
[...]
The sync was successful
data sink of member 1 of type external-demo just disconnected
Main sink of member 1 of type external-demo just disconnected
data sink of member 2 of type file-sync just disconnected
Main sink of member 2 of type file-sync just disconnected
All clients have disconnected
[EXTERNAL-DEMO]: OSyncClient completed.
[1]+ Done ./example_external_demo ~/.config/opensync/0.40/group6/1/pluginpipe
$ 
-------->8-----------

This required changes on the OSyncClient API:

osync_client_{get,set}_{plugin,pipe_path} got added to the public API. Which 
are used to write an external plugin (see reference implementation).

An public interface which shouldn't be widely used yet got changed:
osync_client_run_and_block() got an additional parameter (OSyncError **) and 
changed the return value from void to osync_bool. (Should be very trivial to 
port).

If there are any questions regarding this changes, don't hesistate to ask on 
the opensync-devel Mailinglist.

Thanks to Henrik and Graham for their feedback and help to get this working!

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL * new patch *

[Henrik /K. <he...@ka...>]

Just for reference, the external process basically boils down to this 
function:


static int run_plugin(BZOpenSyncPlugin *pbz) {

    osync_trace(TRACE_ENTRY, "%s", __func__);

    OSyncError *error = NULL;
    OSyncQueue *incoming = NULL;
    OSyncQueue *outgoing = NULL;
    OSyncClient *client = NULL;

    client = osync_client_new(&error);
    if (!client)
        goto done;
   
    osync_client_set_overriding_plugin_dir(client, pbz->mySzPathToLib);

    /* Create connection pipes **/
    incoming = osync_queue_new(pbz->mySzPathToPipe, &error);
    if (!incoming)
        goto error;
   
    if (!osync_queue_create(incoming, &error))
        goto error;
   
    /* We now connect to our incoming queue */
    if (!osync_queue_connect(incoming, OSYNC_QUEUE_RECEIVER, &error))
        goto error;
   
    if (!osync_client_set_incoming_queue(client, incoming, &error))
        goto error;

    osync_queue_unref(incoming);


    osync_client_run_and_block(client);
   
    osync_client_unref(client);

    printf("blueZync synchronization client thread complete\n");

    osync_trace(TRACE_EXIT, "%s", __func__);
    return NS_OK;

 error:
    osync_client_unref(client);
   
 done:   
    osync_trace(TRACE_EXIT_ERROR, "%s: %s", __func__, 
osync_error_print(&error));
    fprintf(stderr, "Unable to initialize environment: %s\n", 
osync_error_print(&error));
    osync_error_unref(&error);
    return NS_ERROR_FAILURE;
}


Henrik /KaarPoSoft wrote:
> Hi all,
>
> I have earlier provided a patch so a plugin of type 
> OSYNC_START_TYPE_EXTERNAL can set 
> osync_plugin_set_start_external_command.
>
> However, after some thought, I actually think it should be on the 
> OSyncPluginConfig.
>
> In this way, different members can have different external commands 
> with the same plugin.
>
> I.e. the config for mozilla-sync plugin could specify thunderbird or 
> sunbird or even the Debian ice-*.
>
> Please find attached a combined patch.
>
> Comments?
>
> /Henrik
>

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL * new patch *

[Henrik /K. <he...@ka...>]

Hi all,

I have earlier provided a patch so a plugin of type 
OSYNC_START_TYPE_EXTERNAL can set osync_plugin_set_start_external_command.

However, after some thought, I actually think it should be on the 
OSyncPluginConfig.

In this way, different members can have different external commands with 
the same plugin.

I.e. the config for mozilla-sync plugin could specify thunderbird or 
sunbird or even the Debian ice-*.

Please find attached a combined patch.

Comments?

/Henrik

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Graham Cobb wrote:
> On Thursday 29 October 2009 15:55:40 Henrik /KaarPoSoft wrote:
>   
>> Daniel Gollub wrote:
>>     
>>> And your real "plugin" which is acutally a complete different process -
>>> or even an own application is implementing a full OSyncClient and
>>> OSyncPlugin environment? Completeyl without that get_sync_info() stuff
>>> ...
>>>       
>> I could do that, but it would take a lot of coding to make my own
>> OSyncClient. That's why I simply use the full opensync library. That's
>> only 10 lines of code!
>>     
>
> I think that is what I have been trying to suggest (badly) -- thanks for the 
> clarification Daniel.
>
> But, of course, Henrik, you shouldn't have to implement your own OSyncClient.  
> What we have to do is to make the client code and plugin support code usable 
> from your process, without bringing in all the rest of OpenSync.  That may be 
> harder than it looks because some functions that plugins call may assume the 
> whole OpenSync environment is present.
>   
Any hints on how to do that would be appreciated!

/Henrik

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Graham Cobb wrote:
> On Thursday 29 October 2009 15:48:33 Henrik /KaarPoSoft wrote:
>   
>> As I wrote in a different thread, I am not sure if my way of
>> implementing the external process is the intended one. I am simply
>> running the same kind of client that would have been running inside
>> osynctool, by calling the same opensync libraries. That means that
>> without osync_client_get_overriding_plugin_dir the opensync library
>> would load the slim mozilla-sync plugin which was intended for
>> osynctool/engine, not the new fat mozilla-sync plugin intended for the
>> external process.
>>     
>
> There seem to be two possible solutions:
>
> 1) The "thin" plugin could just be replaced with a configuration file.  What 
> does OpenSync need to know about external process plugins?  Presumably pretty 
> much just the fact that they are external and how to contact them.
>
> Should we add a config file to supply that info?  Then opensync would just not 
> attempt to load any plugin for which it had already read a config file.
>   
Good idea. Daniel, your call!

> 2) Or, the fat plugin could not really be a plugin.  It could be completely up 
> to the caller of opensync in the external process to have the plugin code 
> loaded (presumably statically linked into the executable) and opensync, when 
> invoked to just create a client, would not load **any** plugins.  There would 
> have to be a function to call to register a plugin by passing in the 
> get_plugin_info address.
>
> I **think** this second option was how the original external code worked, but 
> I haven't checked the code so I might be completely wrong.
>   
Could you point me to where to find "the original external code"?
I have not been able to find any implementation of an external 
process/plugin - except for osplugin.
> I am keen to keep the external client code as minimal as possible.  In 
> particular, if we are every going to make IPC and timeout work properly I 
> certainly don't want two full IPC instances communicating -- there are enough 
> queues as it is!
>   
I don't think this approach introduces any new queues.
I believe there are two bidirectional queues in any case (engine and 
plugin), it is just a question of whether they are accessed from 
different threads in the same thread of from different processes.

/Henrik

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Graham C. <g+o...@co...>]

On Thursday 29 October 2009 15:55:40 Henrik /KaarPoSoft wrote:
> Daniel Gollub wrote:
> > And your real "plugin" which is acutally a complete different process -
> > or even an own application is implementing a full OSyncClient and
> > OSyncPlugin environment? Completeyl without that get_sync_info() stuff
> > ...
>
> I could do that, but it would take a lot of coding to make my own
> OSyncClient. That's why I simply use the full opensync library. That's
> only 10 lines of code!

I think that is what I have been trying to suggest (badly) -- thanks for the 
clarification Daniel.

But, of course, Henrik, you shouldn't have to implement your own OSyncClient.  
What we have to do is to make the client code and plugin support code usable 
from your process, without bringing in all the rest of OpenSync.  That may be 
harder than it looks because some functions that plugins call may assume the 
whole OpenSync environment is present.

Graham

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL

[Graham C. <g+o...@co...>]

On Thursday 29 October 2009 15:48:33 Henrik /KaarPoSoft wrote:
> As I wrote in a different thread, I am not sure if my way of
> implementing the external process is the intended one. I am simply
> running the same kind of client that would have been running inside
> osynctool, by calling the same opensync libraries. That means that
> without osync_client_get_overriding_plugin_dir the opensync library
> would load the slim mozilla-sync plugin which was intended for
> osynctool/engine, not the new fat mozilla-sync plugin intended for the
> external process.

There seem to be two possible solutions:

1) The "thin" plugin could just be replaced with a configuration file.  What 
does OpenSync need to know about external process plugins?  Presumably pretty 
much just the fact that they are external and how to contact them.

Should we add a config file to supply that info?  Then opensync would just not 
attempt to load any plugin for which it had already read a config file.

2) Or, the fat plugin could not really be a plugin.  It could be completely up 
to the caller of opensync in the external process to have the plugin code 
loaded (presumably statically linked into the executable) and opensync, when 
invoked to just create a client, would not load **any** plugins.  There would 
have to be a function to call to register a plugin by passing in the 
get_plugin_info address.

I **think** this second option was how the original external code worked, but 
I haven't checked the code so I might be completely wrong.

I am keen to keep the external client code as minimal as possible.  In 
particular, if we are every going to make IPC and timeout work properly I 
certainly don't want two full IPC instances communicating -- there are enough 
queues as it is!

Graham

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Daniel Gollub wrote:
> On Tuesday 27 October 2009 06:07:50 pm Henrik /KaarPoSoft wrote:
> [...]
>   
>> I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.
>>     
>
> Awesome!
>
> [...]
>   
>> In fact, the engine (e.g. in osynctool) will initialize a proxy, and
>> this proxy will send the name of the plugin and the path to the plugin
>> directory to the external process.
>>     
>
>
> By "plugin directory" you mean the member directory of the group - right?
>
> E.g. ~/.opensync/groupX/Y/
>
> This is at what the engine uses as argument to spawn for proxy. The function 
> osync_client_proxy_spawn() is then creating two pipes in this member 
> directory: pluginpipe and enginepipe
>
>   
No, I do mean the PLUGIN directory.
static osync_bool _osync_client_handle_initialize(OSyncClient *client, 
OSyncMessage *message, OSyncError **error):
    osync_message_read_string(message, &enginepipe, error);
    osync_message_read_string(message, &formatdir, error);
    osync_message_read_string(message, &plugindir, error);
    osync_message_read_string(message, &pluginname, error);
    osync_message_read_string(message, &groupname, error);
    osync_message_read_string(message, &configdir, error);
    osync_message_read_int(message, &haspluginconfig, error);
>
> Why don't you try this:
>
> Just write a very tiny mozilla-sync which just contains the function 
> "get_sync_info"?
>
> It just registers the plugin and set the start-type to 
> OSYNC_START_TYPE_EXTERNAL.
>
> So this plugin (stub) is not using any mozilla API -> no mozilla dependecies 
> .... Engine doesn't load those libraries.
>
>
> The plugin stub only does one thing: register the plugin to the OSyncPluginEnv 
> ... and set the name and description of the plugin, as well as that the plugin 
> is an external one and needs to be started via OSYNC_START_TYPE_EXTERNAL ...
>   
This is exactly what I did.
> And your real "plugin" which is acutally a complete different process - or 
> even an own application is implementing a full OSyncClient and OSyncPlugin 
> environment? Completeyl without that get_sync_info() stuff ...
>   
I could do that, but it would take a lot of coding to make my own 
OSyncClient. That's why I simply use the full opensync library. That's 
only 10 lines of code!

But if you have a proof-of-concept with a full stand-alone OSyncClient 
without to much code, I will certainly have a look!

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Daniel Gollub wrote:
> On Wednesday 28 October 2009 10:04:43 pm Henrik /KaarPoSoft wrote:
>   
>> Please let me know if anybody objects to me checking in the attached
>> combined patch for all the issues I have raised about
>> OSYNC_START_TYPE_EXTERNAL so far.
>>     
>
> I like the command_start addition .. this is somethign which was really 
> missing. Could you explain how you use this for mozilla-sync?
>
> Do you start mozilla if it isn't already started? Or do you trigger the sync 
> via an additional command? Or both?
>   
If you call the "thunderbird" program, it will start a new process if it 
is not already running. I thunderbird is already running, it will simply 
hand off the command line parameters to the already running instance.

The opensync client is triggered by the -mozilla-sync command line 
option which I added to thunderbird.

The sync itself can be triggered by running the engine from osynctool or 
from inside the blueZync plugin to Thunderbird.
> I'm still not convinced why there is need to overwrite the plugindir ... or 
> maybe i didn't understand the problem. Unfortuantely exactly for the new API 
> function to override the path the API documentation is missing why this is 
> needed ...
>
> +OSYNC_EXPORT void osync_client_set_overriding_plugin_dir(OSyncClient *client, 
> char *overriding_plugin_dir);
> +OSYNC_EXPORT char *osync_client_get_overriding_plugin_dir(OSyncClient 
> *client);
> +
>   
As I wrote in a different thread, I am not sure if my way of 
implementing the external process is the intended one. I am simply 
running the same kind of client that would have been running inside 
osynctool, by calling the same opensync libraries. That means that 
without osync_client_get_overriding_plugin_dir the opensync library 
would load the slim mozilla-sync plugin which was intended for 
osynctool/engine, not the new fat mozilla-sync plugin intended for the 
external process.

/Henrik

Re: [Opensync-devel] [RFC] Do not clone-demerge-compare, just compare taking capabilities into account

[Daniel G. <go...@b1...>]

On Saturday 24 October 2009 04:13:33 pm Henrik /KaarPoSoft wrote:
> Dear all,
> 
> Following up on a previous thread on this list:
> >> The current logic in _osync_obj_engine_mapping_find is:
> >> Clone (i.e. make a complete copy of) change1. Demerge fields from the
> >> clone which are not in caps2.
> >> Clone (i.e. make a complete copy of) change2. Demerge fields from the
> >> clone which are not in caps1.
> >> Compare the two cloned objects.
> >> To me this seems unnecessarily complicated and incurs a performance
> >>  penalty. Why is this not implemented as a more "intelligent" compare
> >>  function, which takes capabilities into account?
> >
> > This is pretty new code and more or less just a proof of concept. If you
> > came up with a more efficient solution, let me know. This was the last
> > thing i was working on ... it's not completed.
> 
> OK, here is a suggestion:
> 
> In  _osync_obj_engine_mapping_find in opensync_obj_engine.c we currently
>  have something which boils down to:
> 
> clone_change1 =
>  _osync_obj_engine_clone_and_demerge_change(sinkengine->engine, caps2,
>  change1, error); clone_change2 =
>  _osync_obj_engine_clone_and_demerge_change(sinkengine->engine, caps1,
>  change2, error); change1 = clone_change1;
> change2 = clone_change2;
> tmp_result = osync_change_compare(change1, change2, error);
> 
> This could be replaced with
> 
> tmp_result = osync_change_compare(change1, change2, caps1, caps2, error);
> 
> osync_change_compare and thus osync_data_compare and
>  osync_objformat_compare and maybe other functions would require trivial
>  changes.

Actually i hesistate to make OSyncChange OSyncData more complex ... since you 
could perfeclty use this function without capabilities and stuff like that.

This would als require to break the API.


> 
> It would then boil down to modifying functions like osync_objformat_compare
>  to take capabilities into account (not trivial).
> 
> As far as I can see, this would be a far more efficient solution.


And i don't see the advantage of performance here? You would then end up again 
in calling osync_obj_engine_clone_and_demerge .. .but jsut in a different 
place.

I guess if you want to reduce the numbers of clone and demerges then you need 
to work higher in the stack ... e.g. at the place where 
_osync_obj_engine_clone_and_dmerge gets called today ... and try to demerge 
every change only once and reuse it until a mapping got found.

> 
> The drawback is of course that it needs coding work (=time).
> 
> However, as demerge_xmlformat needs heavy work anyway, I think this may be
>  a good time to consider replacing clone-demerge-compare with a compare
>  taking capabilities into account.
> 
> What do you all think about this proposal?

I guess we should post-pone this ... this is "just" a perforamance 
improvments, which we could still do later without breaking the API. Currently 
we should concentrating in getting things working ASAP, before making them 
fast ... later we can still think about how to make synchronization work with 
O(1) ;)

> 
> /Henrik
> 
> PS: This was about demerge. I have not quite figured merge out, maybe
>  somebody more knowledgeable than me could comment on this...

Merge is currently not active (due to API changes, trivial porting required.)
But the merge process is less complex, since you don't have to deal with 
different combination of capabilities and changes/data.

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [RFC] Using OSYNC_START_TYPE_EXTERNAL

[Daniel G. <go...@b1...>]

On Sunday 25 October 2009 10:59:55 pm Henrik /KaarPoSoft wrote:
> We could add a function called osync_plugin_set_start_external_command 
> in opensync_plugin to be called from get_sync_info:

Ok, thats pretty cool! This is defenitly missing in OpenSync.

> 
> This function could be passed a string in printf format (e.g. 
> "start-external-process %s"), and osync_client_proxy_spawn could call 
> g_strdup_printf with the pipe name as an argument, and execute the 
> resulting command with g_spawn_command_line_async.
> 
> I do not know about other external processes, but with blueZync plugin 
> installed, if you launch "thunderbird --mozilla-sync=/path/to/pipe" it 
> will create a new thunderbird instance if one is not already running, 
> otherwise it will connect to the running thunderbird, which is exactly 
> what we need.
> 
> Comments on this would be most appreciated.

Checked your patch (different thread), looks pretty promosing!

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL

[Daniel G. <go...@b1...>]

On Wednesday 28 October 2009 10:04:43 pm Henrik /KaarPoSoft wrote:
> Dear all,
> 
> To my surprise, I have received only ONE comment (thank you Graham!) to
> my recent postings about OSYNC_START_TYPE_EXTERNAL.

Sorry, i wrote several tests this week - looking forward to some more spare 
time next week. Private things first - you know ;)


> 
> Please let me know if anybody objects to me checking in the attached
> combined patch for all the issues I have raised about
> OSYNC_START_TYPE_EXTERNAL so far.

I like the command_start addition .. this is somethign which was really 
missing. Could you explain how you use this for mozilla-sync?

Do you start mozilla if it isn't already started? Or do you trigger the sync 
via an additional command? Or both?


I'm still not convinced why there is need to overwrite the plugindir ... or 
maybe i didn't understand the problem. Unfortuantely exactly for the new API 
function to override the path the API documentation is missing why this is 
needed ...

+OSYNC_EXPORT void osync_client_set_overriding_plugin_dir(OSyncClient *client, 
char *overriding_plugin_dir);
+OSYNC_EXPORT char *osync_client_get_overriding_plugin_dir(OSyncClient 
*client);
+

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [osynctool] osynctool_list_formats and other functions

[Daniel G. <go...@b1...>]

On Wednesday 28 October 2009 10:21:42 pm Henrik /KaarPoSoft wrote:
> In my opinion there is no relation between trace and return values.

Right, there isn't any.

> 
> Any function which has anything interesting to say should osync_trace.

But this only counts for OpenSync API code ...  you can use what ever you want 
in your OpenSync based application (e.g. osynctool, ...)

You don't have to use osync_trace() in your plugin ... but most OpenSync 
developer or used to osync-trace files already ... (even if no one knows why 
they have those strange file names and why there are soooo many of them ;))

> 
> Any function which could "fail" should
> (1) either return an osync_bool or a pointer which could be checked for 
> NULL.
> (2) have an OSyncError parameter
> (3) set the OSyncError in case it returns false or NULL.
> 

This counts only for OpenSync API ... we don't care about coding style outside 
of the OpenSync API. But i recommend that all OpenSync based application 
should take care about function which could fail and provide a OSyncError**

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [osynctool] osynctool_list_formats and other functions

[Daniel G. <go...@b1...>]

On Wednesday 28 October 2009 08:51:03 pm Jakub Marczyński wrote:
> I started my "devel path" from looking through the osynctool.c
> (http://svn.opensync.org/osynctool/trunk/tools/osynctool.c)

Cool, thanks for joining us! ;)

> 
> And now (at the very beginning) I have some questions:
> 
> 1. Function [line: 164]:
>    static osync_bool osynctool_list_formats(OSyncFormatEnv *env, OSyncError
>  **error)
> 
> Argument "error" is never used in the function. Will it be needed in the
>  future? Do we consider as a bug?

First of all ... osynctool_ is not part of the OpenSync API. It's only 
osynctool code ...

Also keep in mind osynctool is "only" an implementation reference and a tool 
for tests/developers to test the OpenSync API ...

But back to the actual topic ... osynctool_list_formts never calls a function 
which may raise an expection or so which fills a OSyncError struct ...

So actually osynctool could be changed to:

void osynctool_list_formats(OSyncFormatenv *env)


It's not a bug - just obsolete, ugly code in osynctool - not OpenSync ;)



> 
> 2. Some function return "void" and use "osync_trace", while other return
>  "osync_bool" (and do not trace). Are there any specific rules? When do we
>  need trace?

Since osynctool stands for all other application based on the OpenSync API, 
you can use what ever debugging/tracing API you want to use. E.g. use QDebug 
or something else ... outside of OpenSync API code you don't have to use 
osync_trace ...

Same for plugins, you don't have to use osync_trace() .. but you can if you 
like.

The rules for code inside the OpenSync API: there is none.
We usually place osync_trace() in complex function, to have some helpers to 
diagnose bug reports ... if you debug a problem and you think it would be 
handy in future to have some osync_trace() in this function to debug similar 
issues in the future more easily, don't hesistate to add one. But there is no 
requirement that we put on every single function call osync_trace(TRACE_ENTRY) 
osync_trace(TRACE_EXIT) ...

In fact too much osync_trace() calls have bad impact on performance if tracing 
is enabled ... but for that we have also helpers to temporarily disable 
tracing osync_trace_enable() osync_trace_disable() ... e.g. for (quick) loops 
which get called pretty often ...


> "osynctool_list_formats" and "osynctool_list_plugins" seem to be very
>  similar, but they behave differently in this matter.

Right, osync_list_formats should look like osync_list_formats ... if you like 
you can provide a patch for that ;)

> 
> 
> Thanks in advance for answering my questions. I promise there will be some
>  more ;)

Cool - pretty good question. Looking forward to more questions and of course 
to your patches ;)

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Daniel G. <go...@b1...>]

Hi Henrik,


On Tuesday 27 October 2009 06:07:50 pm Henrik /KaarPoSoft wrote:
[...]
> I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.

Awesome!

[...]
> 
> In fact, the engine (e.g. in osynctool) will initialize a proxy, and
> this proxy will send the name of the plugin and the path to the plugin
> directory to the external process.


By "plugin directory" you mean the member directory of the group - right?

E.g. ~/.opensync/groupX/Y/

This is at what the engine uses as argument to spawn for proxy. The function 
osync_client_proxy_spawn() is then creating two pipes in this member 
directory: pluginpipe and enginepipe

> 
> The external process will then set up an identical environment in
> _osync_client_handle_initialize (at least if the external process is
> coded in the simplest possible way I could find: calling
> osync_client_run_and_block and letting the OpenSync libraries take care
> of the rest).

Right.

> 
> This means that the engine (e.g. in osynctool) will load all plugin
> modules, which in turn leads to loading all the libraries the module
> depends on.
> 
> In the mozilla-sync case that means xpcom libraries etc.

Right, but ...

> 
> It also means that the mozilla-sync plugin will need to be compiled the
> Mozilla development files.
> 
> All of this totally unnecesary, as all the Mozilla-relevant stuff is
> taking place in the external process.

.. you don't have to load this.

> 
> The external process will also load all plugin modules - again totally
> unnecessary since it will only need a single plugin.


Why don't you try this:

Just write a very tiny mozilla-sync which just contains the function 
"get_sync_info"?

It just registers the plugin and set the start-type to 
OSYNC_START_TYPE_EXTERNAL.

So this plugin (stub) is not using any mozilla API -> no mozilla dependecies 
.... Engine doesn't load those libraries.


The plugin stub only does one thing: register the plugin to the OSyncPluginEnv 
... and set the name and description of the plugin, as well as that the plugin 
is an external one and needs to be started via OSYNC_START_TYPE_EXTERNAL ...


And your real "plugin" which is acutally a complete different process - or 
even an own application is implementing a full OSyncClient and OSyncPlugin 
environment? Completeyl without that get_sync_info() stuff ...

I'll provide for that a proof-of-concepüt ...

Or do you see already anything why this can't work?

[...]

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] Fixing the OpenSync core development BUS factor

[Daniel G. <go...@b1...>]

On Monday 26 October 2009 11:26:03 am Graham Cobb wrote:
> > Are there any volunteers for following components?
> 
> I wonder if we are going about this the wrong way?  It is extremely logical
>  to  get the core working properly before worrying about the plugins.  But
>  it doesn't seem to be working in motivating developers.

Good point ...

> 
> I think part of the problem is that while the format plugins (in
>  particular)  are broken, no one can use the code they are working on.  I
>  find I am more motivated to work on the engine if I am solving problems
>  which allow me to make progress with my personal goals, and my personal
>  usage.
> 
> How about, now that the engine API is **reasonably** stable, we concentrate
>  on  getting the plugins to work.  Solve some problems like timezones,
>  recurrences, conversions, etc. and get basic sync working for both
>  contacts and events (at least).  Don't even worry about merge/demerge for
>  the moment. That would give us something which, while not releasable to
>  end users, would at least allow the developers to start using it in their
>  daily lives.  The developers would then start to hit the remaining engine
>  problems
> (capabilities, timeouts, IPC deadlocks, conversion paths, mapping, etc.)
>  and  would be more motivated to work on them.


Currently i do only basic testing with contact syncing. No event/todo calendar 
syncing or so ... since event/todo format handling is more complex and has 
still lots of FIXMEs in place ...

But i guess it doesn't change anyhting if i just recommend that concentrate on 
contact syncing -not on calendar syncing for timing being.

I hesistate a bit working on calendar syncing since this is extremly time 
consuming, and also could done by other/new developers which are not so much 
in the core of OpenSync.

Another reason why i'm not concentrating on calendar-syncing is that i still 
think of replacing XMLFormat-plugin with another common-format plugin which is  
based on some PIM-Format-handler library ... something which came up when we 
talked how Synthesis and OpenSync could benefit from each other .. there the 
idea of a libvxx  came up ... maybe something we should look into before we 
start fixing the calendar-format plugins?


> 
> Of course, the danger is that developers do not try to fix the engine
>  problems  if things are working "well enough" for them.  But that can't be
>  worse than the current situation where no progress is happening at
>  all.  And it would encourage new developers -- I don't see how we are
>  going to get new people to join when things are as theoretical as they are
>  now: people want to be able to try things.

Fully agree!

So the bottom-line is that we need to get something working ... so developers 
can play with it and start fixing other stuff. For me this means i need to do 
another bug-screening round and check for bugs which block people for basic 
use ... or could someone else screen the open tickets for such issues, so we 
get re-priorties tickets to work on?

In meanwhile i try to reply to all the techincal question mails ...

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: go...@b1...          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D

Re: [Opensync-devel] [osynctool] osynctool_list_formats and other functions

[Henrik /K. <he...@ka...>]

Hello Jakub!

Great to see a new developer on the project! Welcome!

I am by no means an authoritative source on OpenSync, but I hope my 
comments below will be helpful.

> 1. Function [line: 164]:
>    static osync_bool osynctool_list_formats(OSyncFormatEnv *env, OSyncError **error)
>    
> Argument "error" is never used in the function. Will it be needed in the future? Do we consider as a bug?
>   
No, this is not a bug.
Many functions have an OSyncError** parameter.
Some time in the future the implementation may change  to  something 
which would indeed generate an error.
And we could still keep the API intact.
> 2. Some function return "void" and use "osync_trace", while other return "osync_bool" (and do not trace).
> Are there any specific rules? When do we need trace? 
> "osynctool_list_formats" and "osynctool_list_plugins" seem to be very similar, but they behave differently in this matter.
>   
In my opinion there is no relation between trace and return values.

Any function which has anything interesting to say should osync_trace.

Any function which could "fail" should
(1) either return an osync_bool or a pointer which could be checked for 
NULL.
(2) have an OSyncError parameter
(3) set the OSyncError in case it returns false or NULL.

/Henrik

[Opensync-devel] [RFC][PATCH] OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Dear all,

To my surprise, I have received only ONE comment (thank you Graham!) to 
my recent postings about OSYNC_START_TYPE_EXTERNAL.

Please let me know if anybody objects to me checking in the attached 
combined patch for all the issues I have raised about 
OSYNC_START_TYPE_EXTERNAL so far.

/Henrik

[Opensync-devel] [osynctool] osynctool_list_formats and other functions

[Jakub M. <ja...@o2...>]

I started my "devel path" from looking through the osynctool.c
(http://svn.opensync.org/osynctool/trunk/tools/osynctool.c)

And now (at the very beginning) I have some questions:

1. Function [line: 164]:
   static osync_bool osynctool_list_formats(OSyncFormatEnv *env, OSyncError **error)
   
Argument "error" is never used in the function. Will it be needed in the future? Do we consider as a bug?

2. Some function return "void" and use "osync_trace", while other return "osync_bool" (and do not trace).
Are there any specific rules? When do we need trace? 
"osynctool_list_formats" and "osynctool_list_plugins" seem to be very similar, but they behave differently in this matter.


Thanks in advance for answering my questions. I promise there will be some more ;)

Bloosky
ja...@o2...

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Thanks for your comments, Graham.

Here is a bit of "replies" / new comments.

Graham Cobb wrote:
> Do we need to have the plugin running at all in the engine process?  Can the 
> engine just be told (somehow) to create the proxy and talk to the real plugin 
> in the external process?
>   
If I understand correctly, today the engine is loading up all the plugin 
modules, but for an OSYNC_START_TYPE_EXTERNAL plugin, the engine just 
calls a few functions to get info. So we need "something", that the 
engine can call to know how to deal with the external process. I.e. a 
plugin, but a very lightweight one.
> Similarly, we don't really need the engine (let alone the other plugins) 
> running at all in the external process -- just the other end of the proxy 
> connection.  In fact, I thought I remembered that there was a way to do that 
> (but I would have to look at the code!).  I do remember that when I was 
> looking at the timeout stuff I found the client/proxy stuff confusing!
>   
There might be a different way, but I have not found it. But with very 
little documentation it is kind of hard to look through the whole 
codebase and see if there is a bit that fits is quite hard.
But if there is a better way than what I have proposed I am all ears!
One of the reasons for my mails about this is indeed to see if anyone 
has a better way.
>> So I think we need a mechanism which allows a plugin (of type
>> OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
>> - one for the engine
>> - one for the external process
>>     
>
> Why does it need one for the engine at all?
>   
See above: We need at least a slim plugin to tell the engine how to 
contact the external process.
> The rough way I had thought this would work...
>
> The external plugin is NOT installed in the usual place for Opensync plugins.
>
> The GUI (or osynctool) creates the engine as normal, which loads all the other 
> plugins.
>
> The GUI (or osynctool) calls a special osync function to tell it that there is 
> a remote plugin accessible over such-and-such a pipe. 
I think this info should come from the slim plugin, not the GUI/osynctool.
>  The engine creates a 
> proxy for this.  At this point the engine believes the plugin exists even 
> though there is only a proxy.  It can be added to groups, etc.
>
> Meanwhile, the external process has created the other end of the proxy NOT by 
> creating the whole engine but just by creating a client (is that the right 
> terminology -- I might have to go and look at the code).
>   
Yes, that is also how it would work in my approach. The external process 
just creates a client (but using the opensync lib, which also includes 
the engine etc, which will not be called).
> Either as part of creating that client, or with a separate function, the 
> external process tells the client where to find the plugin (which is not in a 
> standard place) and connects the plugin to the client, with no engine.
Exactly.
>   It 
> may be that the plugin is not even loaded, in the normal opensync meaning of 
> that term -- it could be a static part of the external process and 
> its "get_sync_info" address is passed to the client.
>   
Sure. My reason for making it a "real" opensync plugin was simply that 
then I could use the opensync client code with minimal modifications.
> The external process then runs the client.
>
> I am sure, from your description, that this is not how it actually works 
> today, but this seems easier than having plugin directory overides.
>
>   
First of all it seems that we agree on the overall principles, which is 
great.

The only question is: How to implement it, and here we have slightly 
different approaches.

However, the difference is not very big!

You suggest the GUI tells the engine about the external process; I 
suggest a "slim" plugin.

You suggest the external process loads the "fat" plugin from a different 
location; you suggest basically the same or that no plugin is loaded at all.


Graham, I really appreciate the feedback you have given!

Comments, anybody ?

Re: [Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Graham C. <g+o...@co...>]

On Tuesday 27 October 2009 17:07:50 Henrik /KaarPoSoft wrote:
> Currently both osynctool (or, in a production environment: your own sync
> function in your own GUI) and the external process are using the exact
> same plugin module (such as mozilla-sync).

Henrik,

I haven't looked at the code (maybe a good thing so we can discuss how it 
**should** work).

Do we need to have the plugin running at all in the engine process?  Can the 
engine just be told (somehow) to create the proxy and talk to the real plugin 
in the external process?

> The external process will then set up an identical environment in
> _osync_client_handle_initialize (at least if the external process is
> coded in the simplest possible way I could find: calling
> osync_client_run_and_block and letting the OpenSync libraries take care
> of the rest).

Similarly, we don't really need the engine (let alone the other plugins) 
running at all in the external process -- just the other end of the proxy 
connection.  In fact, I thought I remembered that there was a way to do that 
(but I would have to look at the code!).  I do remember that when I was 
looking at the timeout stuff I found the client/proxy stuff confusing!

> So I think we need a mechanism which allows a plugin (of type
> OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
> - one for the engine
> - one for the external process

Why does it need one for the engine at all?

The rough way I had thought this would work...

The external plugin is NOT installed in the usual place for Opensync plugins.

The GUI (or osynctool) creates the engine as normal, which loads all the other 
plugins.

The GUI (or osynctool) calls a special osync function to tell it that there is 
a remote plugin accessible over such-and-such a pipe.  The engine creates a 
proxy for this.  At this point the engine believes the plugin exists even 
though there is only a proxy.  It can be added to groups, etc.

Meanwhile, the external process has created the other end of the proxy NOT by 
creating the whole engine but just by creating a client (is that the right 
terminology -- I might have to go and look at the code).

Either as part of creating that client, or with a separate function, the 
external process tells the client where to find the plugin (which is not in a 
standard place) and connects the plugin to the client, with no engine.  It 
may be that the plugin is not even loaded, in the normal opensync meaning of 
that term -- it could be a static part of the external process and 
its "get_sync_info" address is passed to the client.

The external process then runs the client.

I am sure, from your description, that this is not how it actually works 
today, but this seems easier than having plugin directory overides.

Graham

[Opensync-devel] How to port a plugin to OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Dear all,

I have created a kind of "proof-of-concept" for porting mozilla-sync to 
use OSYNC_START_TYPE_EXTERNAL.

I post my approach here both to get the comments of the community, and 
in the hope that it may help others.

What I did was actually quite simple, after all:

(1)
In get_sync_info I added
osync_plugin_set_start_type(pOSyncPlugin, OSYNC_START_TYPE_EXTERNAL);

(2)
In the plugin for Thunderbird (i.e *not* mozilla-sync, but the one that 
runs inside Thunderbird) I added the necessary woodo to start up a new 
thread to listen on the plugin pipe.

(3)
I stole the code from osplugin.c and the thread is simply doing this:

 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    OSyncError *error = NULL;
    OSyncQueue *incoming = NULL;
    OSyncQueue *outgoing = NULL;
    OSyncClient *client = NULL;

    client = osync_client_new(&error);
    if (!client)
        goto done;
    
    /* Create connection pipes **/
    incoming = osync_queue_new(pbz->mySzPathToPipe, &error);
    if (!incoming)
        goto error;
    
    if (!osync_queue_create(incoming, &error))
        goto error;
    
    /* We now connect to our incoming queue */
    if (!osync_queue_connect(incoming, OSYNC_QUEUE_RECEIVER, &error))
        goto error;
    
    if (!osync_client_set_incoming_queue(client, incoming, &error))
        goto error;

    osync_queue_unref(incoming);


    osync_client_run_and_block(client);
    
    osync_client_unref(client);
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

As you may have seen in my other mails, there are still a couple of 
outstanding issues.

/Henrik

[Opensync-devel] [RFC] Different libraries for OSYNC_START_TYPE_EXTERNAL

[Henrik /K. <he...@ka...>]

Dear all,

I am working on porting mozilla-sync to use OSYNC_START_TYPE_EXTERNAL.

I already send a mail about this on the list ("[RFC] Using 
OSYNC_START_TYPE_EXTERNAL") on which I still hope to get comments.

However, one more conceptual question has come up, and I would like to 
discuss this with the community.

Currently both osynctool (or, in a production environment: your own sync 
function in your own GUI) and the external process are using the exact 
same plugin module (such as mozilla-sync).

In fact, the engine (e.g. in osynctool) will initialize a proxy, and 
this proxy will send the name of the plugin and the path to the plugin 
directory to the external process.

The external process will then set up an identical environment in  
_osync_client_handle_initialize (at least if the external process is 
coded in the simplest possible way I could find: calling 
osync_client_run_and_block and letting the OpenSync libraries take care 
of the rest).

This means that the engine (e.g. in osynctool) will load all plugin 
modules, which in turn leads to loading all the libraries the module 
depends on.

In the mozilla-sync case that means xpcom libraries etc.

It also means that the mozilla-sync plugin will need to be compiled the 
Mozilla development files.

All of this totally unnecesary, as all the Mozilla-relevant stuff is 
taking place in the external process.

The external process will also load all plugin modules - again totally 
unnecessary since it will only need a single plugin.

So I think we need a mechanism which allows a plugin (of type 
OSYNC_START_TYPE_EXTERNAL) to have TWO different libraries:
- one for the engine
- one for the external process

A very simple way to accomplish this would be:

1)
Add a field to struct OSyncClient:
char *overriding_plugin_dir

2)
Add corresponding getters and setters in osync_client.c

3)
In _osync_client_handle_initialize (which gets passed the OSyncClient)
If overriding_plugin_dir<>NULL then call
osync_plugin_env_load(client->plugin_env, overriding_plugin_dir, error)
instead of
osync_plugin_env_load(client->plugin_env, plugindir, error)

4)
Now the external process can set overriding_plugin_dir after it has 
created the client with osync_client_new.

This would have the additional benefit of keeping the peer-specific (in 
this case: Mozilla-specific) code out of the OpenSync codebase.

So even though Thunderbird 2 and Thunderbird 3 are so different, that it 
seems I will have to create two different plugins, all this would be 
handled inside a plugin for Thunderbird, while keeping the mozilla-sync 
in OpenSync very slim.

Any comments and thoughts on this would be most appreciated.

/Henrik

Re: [Opensync-devel] Fixing the OpenSync core development BUS factor

[<he...@ka...>]

> I wonder if we are going about this the wrong way?  It is extremely
> logical to
> get the core working properly before worrying about the plugins.  But it
> doesn't seem to be working in motivating developers.
>
> I think part of the problem is that while the format plugins (in
> particular)
> are broken, no one can use the code they are working on.  I find I am more
> motivated to work on the engine if I am solving problems which allow me to
> make progress with my personal goals, and my personal usage.
>
> How about, now that the engine API is **reasonably** stable, we
> concentrate on
> getting the plugins to work.  Solve some problems like timezones,
> recurrences, conversions, etc. and get basic sync working for both
> contacts
> and events (at least).  Don't even worry about merge/demerge for the
> moment.
> That would give us something which, while not releasable to end users,
> would
> at least allow the developers to start using it in their daily lives.  The
> developers would then start to hit the remaining engine problems
> (capabilities, timeouts, IPC deadlocks, conversion paths, mapping, etc.)
> and
> would be more motivated to work on them.

I think we need to work in all areas at the same time (engine,
format-plugins, sync-plugins).
If we do not develop engine and format-plugins, the sync-plugins will not
work.
If we do not develop sync-plugins, we will not find and fix the problems
in the engine.

For one particular plugin (mozilla-sync) I can say that without
capabilities, demerger and IPC it simply does not work.

Do not forget that many of the bug reports for the engine comes from
testers and plugin developers hitting a wall, not from a general wish to
finish the loose ends in the engine in itself.

/Henrik

Re: [Opensync-devel] Fixing the OpenSync core development BUS factor

[Graham C. <g+o...@co...>]

On Monday 19 October 2009 17:29:24 Daniel Gollub wrote:
> So how should we solve the problem?
> Should i declare OpenSync as dead? Or should i stop mainting the project
> and hand it over to someone else? If anyone thinks this would help ...
> please let me and the others know. Currently I'm certianly not the best
> choices as a maintainer due to the lack of time i can spent on this
> project. But maybe i still have some understanding how OpenSync in detail
> works, which could be very valuable to finally complete this project ...

If someone with more time wants to aim to take over that might (or might not) 
be a good idea.  But not until they have gathered experience and support by 
making major contributions.  So I don't see any challengers for your position 
for quite a while :-)

> Are there any volunteers for following components?

I wonder if we are going about this the wrong way?  It is extremely logical to 
get the core working properly before worrying about the plugins.  But it 
doesn't seem to be working in motivating developers.  

I think part of the problem is that while the format plugins (in particular) 
are broken, no one can use the code they are working on.  I find I am more 
motivated to work on the engine if I am solving problems which allow me to 
make progress with my personal goals, and my personal usage.

How about, now that the engine API is **reasonably** stable, we concentrate on 
getting the plugins to work.  Solve some problems like timezones, 
recurrences, conversions, etc. and get basic sync working for both contacts 
and events (at least).  Don't even worry about merge/demerge for the moment.  
That would give us something which, while not releasable to end users, would 
at least allow the developers to start using it in their daily lives.  The 
developers would then start to hit the remaining engine problems 
(capabilities, timeouts, IPC deadlocks, conversion paths, mapping, etc.) and 
would be more motivated to work on them.

Of course, the danger is that developers do not try to fix the engine problems 
if things are working "well enough" for them.  But that can't be worse than 
the current situation where no progress is happening at all.  And it would 
encourage new developers -- I don't see how we are going to get new people to 
join when things are as theoretical as they are now: people want to be able 
to try things.

Just an idea -- what do others think?

Graham