Menu
▾
▴
opensync-devel
|
From: Mark E. <ma...@mp...> - 2012-08-01 20:08:44
Attachments:
signature.asc
|
Hi All Thought i'd give this another go. I (try) and maintain the synce project, for syncing to windows mobile. The project has got a bit behind the times, so i'm playing catchup with a few things. I've raised this problem problem before, but really had no time, so i'm trying to give it another go now. We do have a prototype plugin for the 0.40 branch, using the python plugin. However, i get a segfault during the discover phase, in osync_list_copy, with the glib slice allocator returning an evidently invalid address. It's all got a bit beyond me, I'm afraid. I can't see anything obviously wrong, i thought it might be something wierd with glib thread initialisation, but after putting checks in all kinds of places i can't find it. Does anyone have any ideas ? I'm starting to think it'll just be easier to re-write the plugin in C. Mark |
|
From: Chris F. <cd...@fo...> - 2012-08-01 21:31:14
|
On Wed, Aug 01, 2012 at 08:48:29PM +0100, Mark Ellis wrote: > We do have a prototype plugin for the 0.40 branch, using the python > plugin. However, i get a segfault during the discover phase, in > osync_list_copy, with the glib slice allocator returning an evidently > invalid address. > > It's all got a bit beyond me, I'm afraid. I can't see anything obviously > wrong, i thought it might be something wierd with glib thread initialisation, > but after putting checks in all kinds of places i can't find it. Does anyone > have any ideas ? I'm starting to think it'll just be easier to re-write the > plugin in C. I vaguely remember you reporting this before, so I'm glad you're tackling it again! Hopefully I can help a bit more this time. Do you have a backtrace of the segfault? I don't have any windows mobile devices to test with. What would be a good way to reproduce this error without your plugin? Or do you not need a device plugged in, in order to reproduce? Thanks, - Chris |
|
From: Mark E. <ma...@mp...> - 2012-08-02 14:27:26
|
On Wed, 2012-08-01 at 23:31 +0200, Chris Frey wrote: > On Wed, Aug 01, 2012 at 08:48:29PM +0100, Mark Ellis wrote: > > We do have a prototype plugin for the 0.40 branch, using the python > > plugin. However, i get a segfault during the discover phase, in > > osync_list_copy, with the glib slice allocator returning an evidently > > invalid address. > > > > It's all got a bit beyond me, I'm afraid. I can't see anything obviously > > wrong, i thought it might be something wierd with glib thread initialisation, > > but after putting checks in all kinds of places i can't find it. Does anyone > > have any ideas ? I'm starting to think it'll just be easier to re-write the > > plugin in C. > > I vaguely remember you reporting this before, so I'm glad you're tackling > it again! Hopefully I can help a bit more this time. > > Do you have a backtrace of the segfault? > Attached below. > I don't have any windows mobile devices to test with. What would be a > good way to reproduce this error without your plugin? Or do you not need > a device plugged in, in order to reproduce? You can reproduce it wonderfully with the test module included with the python plugin, which is what i have mostly been using, it's a lot simpler to deal with. Also attached 3 patches, api_change.diff is self explanatory, opensync1.diff tweaks it to work against the module built by your git tree, and buildsys.diff fixes it to use your git cmake modules, pass -DCMAKE_MODULE_PATH=<path> to cmake to build it. Haven't applied them in svn 'cos it didn't make much sense to do so, but if you want to import it into git with the patches .... Good job on keeping opensync going by the way ! Mark |
|
From: Mark E. <ma...@mp...> - 2012-08-14 19:04:29
Attachments:
signature.asc
|
On Mon, 2012-08-13 at 19:58 +0200, Chris Frey wrote: > On Mon, Aug 13, 2012 at 03:46:51PM +0100, Mark Ellis wrote: > > Found the problem. In opensync/module/opensync_module.c, the call to > > g_module_open() uses the BIND_LOCAL flag, restricting the symbols from > > the global namespace, which is not used in opensync 0.22. Removing the > > flag resolves my symbol problem. > > > > However, was this flag added for a good reason ? It was added in svn > > revision 4396, and the reasoning makes sense, but I don't know whether > > that was actually a real problem at the time. If the flag is necessary, > > that makes python embedding somewhat difficult. > > Yes, that's definitely there for a reason. It makes it hard to control > what symbols are loaded without the LOCAL flag. > > My personal opinion is that it's much cleaner to use LOCAL... almost like > a namespace at the linker level. And according to Python, namespaces > are a good thing. :-) > > > > The first thing the synce plugin does is import some more python > > modules. The first being dbus, which in turn imports most of it's > > functionality from a C shared object. Quite correctly as I understand > > it, this object is not linked to libpython, and results in a python > > import error from undefined symbols. I first tried passing -Xlinker > > -export-dynamic when linking the python plugin, which didn't help, > > annoying since this seems to be the accepted way of fixing this. > > LD_PRELOAD with libpython fixes it, confirming everything else is ok, > > but the python plugin is linked to libpython, so it should already be > > loaded. > > The undefined symbols... are they only libpython symbols? > > If I import dbus in the sample.py, will I see what you see, or do I need > to actually use it somehow? > Yes you'll see it if you just import dbus. No other unresolved symbols, the python extension objects are linked against all the other libs they need, just not libpython. Mark |
|
From: Chris F. <cd...@fo...> - 2012-08-03 00:24:39
|
On Thu, Aug 02, 2012 at 03:27:04PM +0100, Mark Ellis wrote: > You can reproduce it wonderfully with the test module included with the > python plugin, which is what i have mostly been using, it's a lot > simpler to deal with. Thanks! Mine is crashing in a different place, but it is related to osync_list_copy() / osync_list_free(), and according to my valgrind runs, I'm starting to wonder if somehow Python's malloc system is being used by opensync. The errors I'm getting are pretty weird, and valgrind seems to have lost track of where the memory came from. This is early results, so clarity may come later. :-) Maybe you can run valgrind on yours and see if it gives us any clues? Here's a sample command similar to what I'm using: valgrind --log-file="py-%p.log" --trace-children=yes --leak-check=full --track-origins=yes osynctool --discover pytest > Also attached 3 patches, api_change.diff is self explanatory, > opensync1.diff tweaks it to work against the module built by your git > tree, and buildsys.diff fixes it to use your git cmake modules, pass > -DCMAKE_MODULE_PATH=<path> to cmake to build it. Haven't applied them in > svn 'cos it didn't make much sense to do so, but if you want to import > it into git with the patches .... If I can get this working, I'll create a git repo for the python plugin as well. I've included your api_change.diff and opensync1.diff patches. The buildsys.diff fixes were already in my copy. Thanks, - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-07 21:15:36
|
On Thu, Aug 02, 2012 at 08:24:33PM -0400, Chris Frey wrote: > Thanks! Mine is crashing in a different place, but it is related to > osync_list_copy() / osync_list_free(), and according to my valgrind runs, > I'm starting to wonder if somehow Python's malloc system is being used > by opensync. The errors I'm getting are pretty weird, and valgrind > seems to have lost track of where the memory came from. > > This is early results, so clarity may come later. :-) This is just a status update, in case others are working on this too. One discovery so far today.... I usually set G_SLICE to always-malloc for debugging purposes, but this does not seem to work well with the C Python API for some reason. Memory accesses get really confused. It would be good if I could find a way to split the python_module.c plugin into its own process, I think, but unfortunately, it needs to call into python in order to register the python scripts' plugins. And this happens before initialize. I'm still sensing some odd memory behaviour with Python, but haven't tracked it all down yet. Removing all the _unrefs() from the opensync library python wrapper seems to help a bit though. :-) - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-07 21:41:57
|
On Tue, Aug 07, 2012 at 05:15:30PM -0400, Chris Frey wrote:
> This is just a status update, in case others are working on this too.
Nevermind that previous update. Even with G_SLICE off, and unref()'s
removed, it still crashes, on super basic simple things like
g_string_new("").
Not much clue what the real issue is yet...
- Chris
|
|
From: Chris F. <cd...@fo...> - 2012-08-09 00:13:37
|
On Thu, Aug 02, 2012 at 03:27:04PM +0100, Mark Ellis wrote:
> You can reproduce it wonderfully with the test module included with the
> python plugin, which is what i have mostly been using, it's a lot
> simpler to deal with.
Ahem..... I think this is the problem. :-)
@@ -353,10 +360,9 @@ static void *pm_initialize(OSyncPlugin *plugin, OSyncPluginInfo *info, OSyncErro
Py_DECREF(ret);
/* loop through all objtype sinks, set up function pointers */
sinks = osync_plugin_info_get_objtype_sinks(info);
for (s = sinks; s; s = s->next) {
- sink = (OSyncObjTypeSink *)s;
+ sink = (OSyncObjTypeSink *)s->data;
osync_objtype_sink_set_connect_func(sink, pm_connect);
osync_objtype_sink_set_disconnect_func(sink, pm_disconnect);
Will do more testing, and if it works reliably, I'll create a git repo
for it.
If you could test this and let me know if you find any further problems,
that would be great!
- Chris
|
|
From: Chris F. <cd...@fo...> - 2012-08-14 21:00:58
|
On Tue, Aug 14, 2012 at 08:04:20PM +0100, Mark Ellis wrote: > Yes you'll see it if you just import dbus. No other unresolved symbols, > the python extension objects are linked against all the other libs they > need, just not libpython. Thanks. I think we'll need to convert the python_module into a OSYNC_START_TYPE_EXTERNAL type plugin, but which does not run as its own plugin at all, only a helper. Each python script would also be an external plugin, and be configured by .xml file instead of using the get_sync_info() function callback. The .xml file would reference the python_module program in the <ExternalCommand> element. See the docs/examples/plugins/src/external* files for an example. I think we'd use something like this for the sample.py: <ExternalPlugin version="1"> <Name>python-sample</Name> <LongName>Sample Python plugin</LongName> <Description>Longer description for the sample python plugin.</Description> <ExternalCommand>python_module sample.py %s</ExternalCommand> </ExternalPlugin> This should give each python script module its own forked process space, which might be a good idea. Installing a python module would be a matter of copying the .py script into the usual libopensync1/python-plugins/ directory, and an .xml config into libopensync1/plugins/. The above example would run python_module, which would search under /python-plugins/ for sample.py and use the %s pipe to communicate with opensync. I think this will work. Let me know if you have other ideas, and I'll be implementing it in the meantime. - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-15 18:10:35
|
On Tue, Aug 14, 2012 at 05:00:52PM -0400, Chris Frey wrote: > Thanks. I think we'll need to convert the python_module into a > OSYNC_START_TYPE_EXTERNAL type plugin, but which does not run as its > own plugin at all, only a helper. Ok, this has been implemented. Read the commit message for details. It works with "import dbus" now. :-) - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-09 02:22:51
|
On Wed, Aug 08, 2012 at 08:13:31PM -0400, Chris Frey wrote: > Will do more testing, and if it works reliably, I'll create a git repo > for it. Ok, it's looking better. I'm able to configure two python-sample plugins to sync with each other, doing nothing, reporting no changes. There's a couple issues that need more attention, which are in the git commit history. You can find the latest source here: http://repo.or.cz/w/opensync/python-module.git Give it a whirl and let me know if you get as far as I have. :-) - Chris |
|
From: Mark E. <ma...@mp...> - 2012-08-10 13:10:09
Attachments:
signature.asc
|
On Thu, 2012-08-09 at 04:22 +0200, Chris Frey wrote: > On Wed, Aug 08, 2012 at 08:13:31PM -0400, Chris Frey wrote: > > Will do more testing, and if it works reliably, I'll create a git repo > > for it. > > Ok, it's looking better. I'm able to configure two python-sample plugins > to sync with each other, doing nothing, reporting no changes. There's > a couple issues that need more attention, which are in the git commit > history. > > You can find the latest source here: > > http://repo.or.cz/w/opensync/python-module.git > > Give it a whirl and let me know if you get as far as I have. :-) > > - Chris > Chris, you're a legend ! Isn't it great what a fresh pair of eyes can see. Now I can start getting to grips with this properly. I'd also noticed, like you, that the no config route isn't very successful, so I guess for know a dummy config will do. Thanks ! Mark |
|
From: Mark E. <ma...@mp...> - 2012-08-20 10:30:46
|
On Wed, 2012-08-15 at 20:10 +0200, Chris Frey wrote: > On Tue, Aug 14, 2012 at 05:00:52PM -0400, Chris Frey wrote: > > Thanks. I think we'll need to convert the python_module into a > > OSYNC_START_TYPE_EXTERNAL type plugin, but which does not run as its > > own plugin at all, only a helper. > > Ok, this has been implemented. Read the commit message for details. > > It works with "import dbus" now. :-) > Yep, it certainly does, thanks again. Attached a couple of patches, fixes exception reporting from python and thread state after initialisation. Getting there now, the sync process completes with the synce plugin, hurrah ! Some internal formatting problems still to sort, but there is definitely light at the end of the tunnel. Mark |
|
From: Chris F. <cd...@fo...> - 2012-08-20 19:03:36
|
On Mon, Aug 20, 2012 at 11:30:19AM +0100, Mark Ellis wrote: > Attached a couple of patches, fixes exception reporting from python and > thread state after initialisation. Thanks! Applied. - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-10 17:40:42
|
On Fri, Aug 10, 2012 at 02:09:59PM +0100, Mark Ellis wrote: > Chris, you're a legend ! Isn't it great what a fresh pair of eyes can > see. > > Now I can start getting to grips with this properly. I'd also noticed, > like you, that the no config route isn't very successful, so I guess for > know a dummy config will do. Thanks ! Glad to hear you're having success! I'm still working out small details in the actual sync part. Data flows from file-sync -> python-sample, but not quiet in the other direction yet. I think there are parts of the opensync API still missing from the python wrappers. If you run into bits of API that you need, let me know, or send patches. :-) - Chris |
|
From: Chris F. <cd...@fo...> - 2012-08-10 18:07:42
|
On Fri, Aug 10, 2012 at 01:40:35PM -0400, Chris Frey wrote: > Glad to hear you're having success! I'm still working out small details > in the actual sync part. Data flows from file-sync -> python-sample, but > not quiet in the other direction yet. Ok, data flows in both directions now, and the slow_sync flag argument in the get_changes callback has been implemented. - Chris |
|
From: Mark E. <ma...@mp...> - 2012-08-13 10:16:33
Attachments:
signature.asc
|
On Fri, 2012-08-10 at 20:07 +0200, Chris Frey wrote: > On Fri, Aug 10, 2012 at 01:40:35PM -0400, Chris Frey wrote: > > Glad to hear you're having success! I'm still working out small details > > in the actual sync part. Data flows from file-sync -> python-sample, but > > not quiet in the other direction yet. > > Ok, data flows in both directions now, and the slow_sync flag argument > in the get_changes callback has been implemented. > > - Chris > You're doing a magnificent job on this, many thanks. I've got some old patches to extend the wrapper, i'll have a look through them and see what state they're in a some point. I've hit a snag early on, i think this may be one for a python list, but see what you think. The first thing the synce plugin does is import some more python modules. The first being dbus, which in turn imports most of it's functionality from a C shared object. Quite correctly as I understand it, this object is not linked to libpython, and results in a python import error from undefined symbols. I first tried passing -Xlinker -export-dynamic when linking the python plugin, which didn't help, annoying since this seems to be the accepted way of fixing this. LD_PRELOAD with libpython fixes it, confirming everything else is ok, but the python plugin is linked to libpython, so it should already be loaded. And to add insult to injury, this never happened in the 0.22 plugin. Any thoughts ? Mark |
|
From: Mark E. <ma...@mp...> - 2012-08-13 14:47:19
Attachments:
signature.asc
|
On Mon, 2012-08-13 at 12:16 +0200, Mark Ellis wrote: > On Fri, 2012-08-10 at 20:07 +0200, Chris Frey wrote: > > On Fri, Aug 10, 2012 at 01:40:35PM -0400, Chris Frey wrote: > > > Glad to hear you're having success! I'm still working out small details > > > in the actual sync part. Data flows from file-sync -> python-sample, but > > > not quiet in the other direction yet. > > > > Ok, data flows in both directions now, and the slow_sync flag argument > > in the get_changes callback has been implemented. > > > > - Chris > > > > You're doing a magnificent job on this, many thanks. I've got some old > patches to extend the wrapper, i'll have a look through them and see > what state they're in a some point. > > I've hit a snag early on, i think this may be one for a python list, but > see what you think. > > The first thing the synce plugin does is import some more python > modules. The first being dbus, which in turn imports most of it's > functionality from a C shared object. Quite correctly as I understand > it, this object is not linked to libpython, and results in a python > import error from undefined symbols. I first tried passing -Xlinker > -export-dynamic when linking the python plugin, which didn't help, > annoying since this seems to be the accepted way of fixing this. > LD_PRELOAD with libpython fixes it, confirming everything else is ok, > but the python plugin is linked to libpython, so it should already be > loaded. > > And to add insult to injury, this never happened in the 0.22 plugin. > Found the problem. In opensync/module/opensync_module.c, the call to g_module_open() uses the BIND_LOCAL flag, restricting the symbols from the global namespace, which is not used in opensync 0.22. Removing the flag resolves my symbol problem. However, was this flag added for a good reason ? It was added in svn revision 4396, and the reasoning makes sense, but I don't know whether that was actually a real problem at the time. If the flag is necessary, that makes python embedding somewhat difficult. Mark |
|
From: Chris F. <cd...@fo...> - 2012-08-13 17:58:43
|
On Mon, Aug 13, 2012 at 03:46:51PM +0100, Mark Ellis wrote: > Found the problem. In opensync/module/opensync_module.c, the call to > g_module_open() uses the BIND_LOCAL flag, restricting the symbols from > the global namespace, which is not used in opensync 0.22. Removing the > flag resolves my symbol problem. > > However, was this flag added for a good reason ? It was added in svn > revision 4396, and the reasoning makes sense, but I don't know whether > that was actually a real problem at the time. If the flag is necessary, > that makes python embedding somewhat difficult. Yes, that's definitely there for a reason. It makes it hard to control what symbols are loaded without the LOCAL flag. My personal opinion is that it's much cleaner to use LOCAL... almost like a namespace at the linker level. And according to Python, namespaces are a good thing. :-) > The first thing the synce plugin does is import some more python > modules. The first being dbus, which in turn imports most of it's > functionality from a C shared object. Quite correctly as I understand > it, this object is not linked to libpython, and results in a python > import error from undefined symbols. I first tried passing -Xlinker > -export-dynamic when linking the python plugin, which didn't help, > annoying since this seems to be the accepted way of fixing this. > LD_PRELOAD with libpython fixes it, confirming everything else is ok, > but the python plugin is linked to libpython, so it should already be > loaded. The undefined symbols... are they only libpython symbols? If I import dbus in the sample.py, will I see what you see, or do I need to actually use it somehow? Thanks, - Chris |
Thanks for helping keep SourceForge clean.
X