tipsyTOO Files

Manual python module synchronize
================================
Created by: Diethard Jansen, 27-10-2010

This version 0.1.0, 27-4-2012
Added unit tests and repaired some problems that were found during unittesting.

Original idea of using file indexes for synchronisation: Jacob van Dijk.

"A tiny python module to do massive synchronisation jobs."

De python module synchronize can be used for following jobs:
    1) Online synchronisation
    2) Offline synchronisation 
    3) Timeline synchronisation

ad 1)
'Normal' synchronisation tools use online synchronisation.

ad 2)
This particular tool can also synchronize remote sites.
For this it uses "file index files" which can be send by email.
The received result can be used to compare with a locally build
file index which will result in an archive that contains the
differences and that can be send back to remote site.

Ad 3)
Of one source directory different snapshots can be taken at
different times using "file index files".
See following timeline where at different times file indexes have
been created at different times.

----1-------2--------3--------4------> time

At time 1, 2, 3 and 4  a file index file has been created.

Those file index files can be compared to see what has changed to
a directory over time. The difference can be delivered
when neccesary.

At time 1 source dir is copied and delivered to remote site.
At any time after 2, 3 or  4 an archive can be created and send
to synchronize remote site using created file indexes.

An example case taken from a Water company.
All fitters carry a laptop with stored on the laptop a directory
that contains about 200.000 sketches of houseconnections. 
Just after they receive the sketches a file index is created.
After some time they login again to synchronize their locally
stored sketches from the centrally stored library of sketches.
Using the timed file index files a relative small archive of
changes can be created and used to update the locally stored
directory of sketches.

Usage of module synchronize in python script:

1 online synchronisation
========================
import synchronize
s = synchronize.synchronizer()
s.sourceDir = "C:\\data\\pictures"
s.targetDir = "E:\\backup\\pictures"
s.synchronize()

# or using less code:

import synchronize
s = synchronize.synchronizer()
s.synchronize("C:\\data\\pictures", "E:\\backup\\pictures")

2 offline synchronisation
=========================
# step 1 create remote index file of remote file that needs to
# be synchronized.

import synchronize
s = synchronize.synchronizer()
l_fileIndex = s.fileIndex("D:\\pictures")
s.storeObjectToFile(l_fileIndex, "D:\\pictures.idx")

# step 2 create at source location an archive to deliver

import synchronize
s = synchronize.synchronizer()
l_fileIndex = s.restoreObjectFromFile("C:\\temp\\pictures.idx")
s.sourceDir = "C:\\data\\pictures"
s.targetIndex = l_fileIndex
s.createArchive("C:\\data\\picture_archive")
# Compress the created map holding archive and send to remote location 

# step 3 synchronize remote directory with received and unpacked archive

import synchronize
s = synchronize.synchronizer()
s.targetDir = "D:\\data\pictures"
# add optional argument p_delete to stop unintentially removing
# pictures person has added at remote location
# in this case the person receiving pictures has decided not
# to have any pictures deleted at his side (add only).
s.synchonizeUsingArchive("C:\\temp\\picture_archive", p_delete=False):

3 timeline synchronisation
==========================
# step 1 create file index at a date.

import synchronize
s = synchronize.synchronizer()
l_fileIndex = s.fileIndex("C:\\data\\pictures")
s.storeObjectToFile(l_fileIndex, "C:\\pictures20100801.idx")

# step 2, create file index at a later date 
import synchronize
s = synchronize.synchronizer()
l_fileIndex = s.fileIndex("C:\\data\\pictures")
s.storeObjectToFile(l_fileIndex, "C:\\pictures20100815.idx")

# step 3
# Configure synchronizer to use compare on last mutation date
# instead of filesize when comparing files with identical
# names.

import synchronize
s = synchronize.synchronizer()
# default value = False, using comparison on filesize
s.useTime2Compare = True
l_firstFileIndex = s.restoreObjectFromFile("C:\\pictures20100801.idx")
l_lastFileIndex = s.restoreObjectFromFile("C:\\pictures20100815.idx")
s.sourceIndex = l_lastFileIndex
s.targetIndex = l_firstFileIndex
s.targetDir = ("D:\\data\pictures")
s.synchronize()

or for offline purposes.. replace last two statements with:
s.createArchive("C:\\data\\picture_archive20100815")

and send compressed result to person who can use archive to synchronize
target directory.