jfilesync-news — JFileSync News

Re: [Jfilesync-news] memory usage

[Jens H. <hei...@in...>]

Hi Stefan:

Unfortunately, there is currently no solution to this problem. As Erich
already said, you may try to decrease the amount of files compared, or
increase the memory, Java is allowed to use, as you did.

The problem is, that the complete hierarchy of files on source and
target side is held in memory after comparison. It future, it would be
better to store this information somewhere on the hard disk and stream
to a file during comparison and read from the file when browsing the
compared files and when doing the synchronization. Moreover, it will
help avoiding referencing the Java File object and keeping it for all
files compared. However, all this will cause significant refactoring and
is no short-term task. I'll try to have a closer look at it beginning of
next year.

As to your second question, you may specify an exclude filter for your
directory before comparison, or alternatively, de-select the directory
pair after comparison from the comparison table (this will of course
have no effect in terms of memory usage). It is not possible to
de-select the file pair from the profile manager.


Cheers,

Jens.


Erich Musick wrote:
> I believe I've encountered a similar problem in the past, though I can't
> recall if I got the Out of Memory error or whether the program just locked
> up. I know with certainty that I've observed the latter.
> 
> I was syncing several directories that were each fairly large and contained
> a lot of files. My solution was to make a couple of different profiles and
> run each separately. It's kind of annoying, but it gets the job done.
> 
> Interestingly, it seemed to have more of a tendency to lock up on OSX than
> on my XP Home install. Both PCs have 1GB ram. I think the cause of this
> might have been that I was backing up my entire Library folder, which
> consists of far too many files.
> 
> Hope this information helps.
> 
> Erich Musick
> http://erichmusick.com
> 
> 
> -----Original Message-----
> From: jfi...@li...
> [mailto:jfi...@li...] On Behalf Of Stefan
> Hermanek
> Sent: Wednesday, December 12, 2007 11:52 AM
> To: jfi...@li...
> Subject: [Jfilesync-news] memory usage
> 
> i tried JFileSync last day and this is what i experienced:
> 
> windows2000, sp4, 1gb ram, java 1.5
> 
> source directory: 167.000 files
> target directory: already 164.000 files from an old backup of the source
> 
> i started JFileSync with java -Xmx256m -jar \lib\jfsync.jar
> 
> aprox. 4000 files were deleted in the target directory and 5000 new files
> copied to it.
> one second after the last copied file JFileSyn termineated with a
> OutOfMemoryError: java heap space.
> the task manager showed 370mb of memory for the java process.
> 
> then i tried to synchronize a directory of aprx. 36.000 files with an
> empty target directory: java used 370mb of memory. why is so much memory
> needed just for copying files?
> 
> then i did another synchronization with 10.000 files to an empty
> directory: used memory raised to 600mb (i don't know how java can do this
> with the -Xmx256m option?).
> after some repeated compare actions of the just synchronized directories
> java ended up with 656mb and another OutOfMemoryError.
> 
> i like JFileSync because of its clear user interface, but is there a way
> to get along with such a number of files and a reasonable memory usage?
> 
> btw, is it possible to exclude a directory pair from beeing
> compared/synchronized in a profile without deleting it?
> 
> thanx, stefan
> 
> 
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> JFileSync-News mailing list
> JFi...@li...
> https://lists.sourceforge.net/lists/listinfo/jfilesync-news
> 
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> JFileSync-News mailing list
> JFi...@li...
> https://lists.sourceforge.net/lists/listinfo/jfilesync-news

Re: [Jfilesync-news] memory usage

[Erich M. <ma...@er...>]

I believe I've encountered a similar problem in the past, though I can't
recall if I got the Out of Memory error or whether the program just locked
up. I know with certainty that I've observed the latter.

I was syncing several directories that were each fairly large and contained
a lot of files. My solution was to make a couple of different profiles and
run each separately. It's kind of annoying, but it gets the job done.

Interestingly, it seemed to have more of a tendency to lock up on OSX than
on my XP Home install. Both PCs have 1GB ram. I think the cause of this
might have been that I was backing up my entire Library folder, which
consists of far too many files.

Hope this information helps.

Erich Musick
http://erichmusick.com


-----Original Message-----
From: jfi...@li...
[mailto:jfi...@li...] On Behalf Of Stefan
Hermanek
Sent: Wednesday, December 12, 2007 11:52 AM
To: jfi...@li...
Subject: [Jfilesync-news] memory usage

i tried JFileSync last day and this is what i experienced:

windows2000, sp4, 1gb ram, java 1.5

source directory: 167.000 files
target directory: already 164.000 files from an old backup of the source

i started JFileSync with java -Xmx256m -jar \lib\jfsync.jar

aprox. 4000 files were deleted in the target directory and 5000 new files
copied to it.
one second after the last copied file JFileSyn termineated with a
OutOfMemoryError: java heap space.
the task manager showed 370mb of memory for the java process.

then i tried to synchronize a directory of aprx. 36.000 files with an
empty target directory: java used 370mb of memory. why is so much memory
needed just for copying files?

then i did another synchronization with 10.000 files to an empty
directory: used memory raised to 600mb (i don't know how java can do this
with the -Xmx256m option?).
after some repeated compare actions of the just synchronized directories
java ended up with 656mb and another OutOfMemoryError.

i like JFileSync because of its clear user interface, but is there a way
to get along with such a number of files and a reasonable memory usage?

btw, is it possible to exclude a directory pair from beeing
compared/synchronized in a profile without deleting it?

thanx, stefan



-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
JFileSync-News mailing list
JFi...@li...
https://lists.sourceforge.net/lists/listinfo/jfilesync-news

[Jfilesync-news] memory usage

[Stefan H. <her...@ut...>]

i tried JFileSync last day and this is what i experienced:

windows2000, sp4, 1gb ram, java 1.5

source directory: 167.000 files
target directory: already 164.000 files from an old backup of the source

i started JFileSync with java -Xmx256m -jar \lib\jfsync.jar

aprox. 4000 files were deleted in the target directory and 5000 new files
copied to it.
one second after the last copied file JFileSyn termineated with a
OutOfMemoryError: java heap space.
the task manager showed 370mb of memory for the java process.

then i tried to synchronize a directory of aprx. 36.000 files with an
empty target directory: java used 370mb of memory. why is so much memory
needed just for copying files?

then i did another synchronization with 10.000 files to an empty
directory: used memory raised to 600mb (i don't know how java can do this
with the -Xmx256m option?).
after some repeated compare actions of the just synchronized directories
java ended up with 656mb and another OutOfMemoryError.

i like JFileSync because of its clear user interface, but is there a way
to get along with such a number of files and a reasonable memory usage?

btw, is it possible to exclude a directory pair from beeing
compared/synchronized in a profile without deleting it?

thanx, stefan

[Jfilesync-news] no synchronization

[Jordan M. <vos...@ho...>]

Hi,
=20
I downloaded JFileSync 2.2, installed it in /opt folder on a Fedora 7 (a Th=
inkpad T30) system.
I tested a synchonization between 2 local folders /root/Publique (empty) an=
d /root/T=E9l=E9chargement (with files into).
=20
Configuration : Merge structure, nothing as include or exclude.
=20
Comparison seems to work, since JFilSync list the files correctly but when =
i hit Synchronization button, nothing happens. The files are not copied.
=20
Any idea why ?
=20
NB : I use Java 1.6.0 from sun (installed in /opt).
=20
Thanks in advance
=20
Dee
_________________________________________________________________
Essayez Live.com, votre nouvelle page d'accueil ! Personnalisez-la en quelq=
ues clics pour retrouver tout ce qui vous int=E9resse au m=EAme endroit.
http://www.live.com/getstarted=

[Jfilesync-news] Does JFileSync support FTP to sync the two directories?

[Christopher D. J. <chr...@gm...>]

If it does, where do I configure it?

If it does not, then is it planned to be added?

Chris

Re: [Jfilesync-news] review of JFileSync

[plastic r. <pla...@te...>]

----- Original Message -----
From: "Jens Heidrich" <jhe...@rh...>
To: "plastic rat" <pla...@te...>
Subject: Re: [Jfilesync-news] review of JFileSync
Date: Thu, 13 Oct 2005 18:07:27 +0200

>=20
> Hi Tom:
>=20
> First of all, thanks for your detailed review of JFS :-)
thank you for writing JFileSync :-)
>=20
> plastic rat wrote:
> > You may be interested to know I have written a review of=20
> > JFileSync and a comparison with other file synchronizers, it is=20
> > at=20
> > http://www.tomkelsey.pwp.blueyonder.co.uk/projects/synchronizer_review.=
html,=20
> > please reply with any comments.
>=20
> Here are some comments to some features requested and our plans for the
> future:
>=20
> A) 'token' mode
> Yes, that's an interesting feature and on our my todo list for some time
> ;-). The original plan was to create a kind of patch package you apply
> to a certain file system in order to sync it with another one. The
> package should be created based on an XML representation of the file
> system structure you want to synchronize (like the current
> synchronization history) and contains a set of commands (files to copy
> and delete) and the corresponding files that would have to be modified.
>=20

OK, thats a good idea. If the xml file is stored on the removable disk with=
 the patch, then that is equivalant to 'token mode'.

To get this to match the 'token' mode completely you would need to have JFS=
 detect when a directory had a patch in it and:
 1 apply the patch to the other directory, and then delete the patch
 2 generate a new patch based on the other directory and put this in the xm=
l file directory
 3 update the xml file to match the other dir
ideally this would all happen automatically when you tried to 'sync' a dire=
ctory that contained a patch

> May be, I have some time to work on it for 2.2, but it's unlikely,
> because current effort is more focused on JFS usability issues.
>=20
> B) client-server compressed protocol
> C) client-server encrypted protocol
> Yes, JFS has a built-in server that doesn't support compression nor
> encryption. The usage scenario is to use this server as part of an SSH
> tunnel (which usually is able to support this kind of stuff). I'm
> thinking of directly supporting connections to an SFTP server, but this
> would limit server client connections to Unix (and Cygwin-based)
> systems. It should also be easier to use the server from the GUI in the
> future. For encrypten we would have to use SSLSockets, which would
> perfectly be possible, but nobody demanded it so far ;-).
You're right encryption should probably not be done in a syncing program.
Compression may be more valuable. Unison and rsync both claim they only sen=
d the parts of files that have changed. I guess they must do this by partio=
ning the file and checksumming each part. This would use less bandwidth tha=
n running it through a compressed link.

>=20
> D) >2 machines
> JFS stores a history for each file pair; so a source directory can be
> synchronized with multiple target directories (or vice versa ;-).
> However, if you connect an external drive that uses drive letters (e.g.,
> "D:") you will have problems because the file pairs look the same even
> if different external drives will be connected to your system
> subsequently (always e.g., src=3D"c:\MyData", tgt=3D"d:\MyData"). If you
> access multiple network drives, for instance, there should be no problem
> with mixing up histories (e.g., pair 1: src=3D"c:\MyData",
> tgt=3D"\\test1\MyData" and pair 2: src=3D"c:\MyData", tgt=3D"\\test2\MyDa=
ta").
>=20
ok. i didn't know this, I will update the review.
The situation of different removable drives could be detected maybe by stor=
ing a GUID in each directory that is synced, and a copy of it in the synchr=
onisation history?

> E) select single files
> Yes, only directories can be part of sync profiles.
>=20
> F) use non RW media
> Yes, it is only supported by the underlying system drivers (if your OS
> is able to integrate the CD-R as a standard drive that can be accessed
> like any other folder).
>=20
I put in this criteria because I had a problem with this before got I got a=
 usb drive.
Namely one of the machines I was working on could not read packet formatted=
 cds, but all the syncing progs available needed to write directly to the d=
irectory.
so I had to:
write one directory onto the cd
move the cd to other machine
do a merge on all files
write the directory on this machine onto cd
repeat on other machine

This could be solved by your patch feature.
Basically you could use JFS to generate a patch based on an xml file on the=
 cd, then write the patch & new xml file to a new session on the cd.

> G) dummy mode
> JFS only supports a "test mode" (via Profile Manager -> Advanced), it
> will only simulate a synchronization, but it will, however, not output
> the files that have to be copied and deleted, but you may view these
> files anyway by pressing "Show Details" before starting a synchronization.
>=20
I was thinking of getting an output on the command line so you could implem=
ent features like creating patches in a separate script
>=20
> Thanks again for your review and for evaluating JFileSync :-).
>=20
no problem.
thanks again for writing JFileSync

>=20
> Cheers,
>=20
> Jens.


bye,

Tom


--=20
___________________________________________________
Play 100s of games for FREE! http://games.mail.com/

Re: [Jfilesync-news] review of JFileSync

[Jens H. <jhe...@rh...>]

Hi Tom:

First of all, thanks for your detailed review of JFS :-)

plastic rat wrote:
> You may be interested to know I have written a review of JFileSync and a comparison with other file synchronizers, it is at http://www.tomkelsey.pwp.blueyonder.co.uk/projects/synchronizer_review.html, please reply with any comments.

Here are some comments to some features requested and our plans for the
future:

A) 'token' mode
Yes, that's an interesting feature and on our my todo list for some time
;-). The original plan was to create a kind of patch package you apply
to a certain file system in order to sync it with another one. The
package should be created based on an XML representation of the file
system structure you want to synchronize (like the current
synchronization history) and contains a set of commands (files to copy
and delete) and the corresponding files that would have to be modified.

May be, I have some time to work on it for 2.2, but it's unlikely,
because current effort is more focused on JFS usability issues.

B) client-server compressed protocol
C) client-server encrypted protocol
Yes, JFS has a built-in server that doesn't support compression nor
encryption. The usage scenario is to use this server as part of an SSH
tunnel (which usually is able to support this kind of stuff). I'm
thinking of directly supporting connections to an SFTP server, but this
would limit server client connections to Unix (and Cygwin-based)
systems. It should also be easier to use the server from the GUI in the
future. For encrypten we would have to use SSLSockets, which would
perfectly be possible, but nobody demanded it so far ;-).

D) >2 machines
JFS stores a history for each file pair; so a source directory can be
synchronized with multiple target directories (or vice versa ;-).
However, if you connect an external drive that uses drive letters (e.g.,
"D:") you will have problems because the file pairs look the same even
if different external drives will be connected to your system
subsequently (always e.g., src="c:\MyData", tgt="d:\MyData"). If you
access multiple network drives, for instance, there should be no problem
with mixing up histories (e.g., pair 1: src="c:\MyData",
tgt="\\test1\MyData" and pair 2: src="c:\MyData", tgt="\\test2\MyData").

E) select single files
Yes, only directories can be part of sync profiles.

F) use non RW media
Yes, it is only supported by the underlying system drivers (if your OS
is able to integrate the CD-R as a standard drive that can be accessed
like any other folder).

G) dummy mode
JFS only supports a "test mode" (via Profile Manager -> Advanced), it
will only simulate a synchronization, but it will, however, not output
the files that have to be copied and deleted, but you may view these
files anyway by pressing "Show Details" before starting a synchronization.


Thanks again for your review and for evaluating JFileSync :-).


Cheers,

Jens.

[Jfilesync-news] review of JFileSync

[plastic r. <pla...@te...>]

You may be interested to know I have written a review of JFileSync and a co=
mparison with other file synchronizers, it is at http://www.tomkelsey.pwp.b=
lueyonder.co.uk/projects/synchronizer_review.html, please reply with any co=
mments.

regards
Tom

--=20
___________________________________________________
Play 100s of games for FREE! http://games.mail.com/

[Jfilesync-news] JFileSync Version 2.1 released

[Jens H. <hei...@in...>]

JFileSync Version 2.1 is released...

You can get it via the JFS Download Pages:
http://jfilesync.sourceforge.net

JFS 2.1 allows you to change the action performed for selected file
pairs of the synchronization table via a popup menu. JFS displays a
warning in case of not stored profile settings and three new views on
the main action table were added. Furthermore, this release corrects a
problem related to the management of synchronization histories.


Cheers,

Jens Heidrich.

[Jfilesync-news] JFileSync Version 2.0.1 released

[Jens H. <hei...@in...>]

JFileSync Version 2.0.1 released...

Please have a look at the updated JFileSync homepage:
http://jfilesync.sourceforge.net
in order to get further details about the lastest release.


Cheers,

Jens Heidrich.

[Jfilesync-news] JFileSync Version 1.6 released...

[Jens H. <hei...@in...>]

After quite a long time ;-)
JFileSync Version 1.6 released...

JFileSync 1.6 comes with two more synchronization modes,
some usability improvements, and provides a more efficient
JFS server implementation.

Please read the release notes and the change log and have
a look at the screenshots and features section to get further
details about the lastest version:
http://jfilesync.sourceforge.net


Cheers,

Jens Heidrich.

[Jfilesync-news] (no subject)

[Heidrich, J. <jen...@ie...>]

JFileSync Version 1.5 released...

JFS 1.5 includes a small server application which allows for
synchronizing with directory structures not direcly available
from your local system.

Please read the release notes and the change log and have
a look at the screenshots and features section to get further
details about the lastest version:
http://jfilesync.sourceforge.net


Regards,

Jens Heidrich.

[Jfilesync-news] News Archives unavailable.

[Richard G. <ric...@ti...>]

Hi!

The JFileSync-News Archives 
<http://sourceforge.net/mailarchive/forum.php?forum=jfilesync-news>.
is unavailable.

http://sourceforge.net/mailarchive/forum.php?forum=jfilesync-news

Thanks,

Richard G.