#30 Memory Issues

[Anonymous]

Hi everybody,

I wanted to sync 2 Plone Sites witz ZSyncer, but it allocates too much memory when I try to sync a full folder. I made tests with little amount of files in folders - no problem, but if I try bigger ones (>100mb) I get memory errors.

Top tells me that python is using all memory - so I created a new swap file with 5gb - nothing changed - ZSyncer always tolds me:

Thu Sep 13 14:20:03 2007 ERROR pulling dc Traceback (most recent call last):

File "/var/lib/zope2.9/instance/plone-site/Products/ZSyncer/ZSyncer.py", line 375,
in manage_pullFromRemote
data = serverconn.manage_getExportData(obj_path)

File "/var/lib/zope2.9/instance/plone-site/Products/ZSyncer/ConnectionMgr.py", lin
e 167, in __call__
raise rd[0], rd[1]

MemoryError: out of memory

Is there anything I can do to lower the memory consumption? I only got 1GB and 6GB Swap at the moment ...

Chris

[Nobody/Anonymous]

Logged In: NO

To contact me:
ICQ: 374150447

[Paul Winkler]

Logged In: YES
user_id=38329
Originator: NO

Some questions:

What ZSyncer version are you using? Are you using the same version on both zope instances?

From the traceback it looks like you're pulling from the remote server to the local one.
(where "local" just means the one that you're viewing the UI on.)
Can you confirm that?

Which instance is running out of memory - "local" or "remote"? Or both?

Are you using the ZMI or ZSyncerTool via the Plone UI? (Maybe not relevant, I don't know if the basic example skin works in Plone, I haven't tried.)

Can you try using the Export tab in the ZMI to save the relevant folder to a .zexp file and tell me how big that file is?
There are several places where ZSyncer naively uses binary strings to hold data in memory, when it should be using temporary files.
But I'm surprised that a folder you estimated at 100 MB uses several GB of RAM.

Can you check your ulimit? As the user that runs your zope process, do this: ulimit -a
and show me the output. Thanks.