Menu
▾
▴
safekeep-commit — Source code changes (CVS commits) for SafeKeep
You can subscribe to this list here.
| 2007 |
Jan
(76) |
Feb
(76) |
Mar
(54) |
Apr
(14) |
May
(23) |
Jun
(21) |
Jul
|
Aug
|
Sep
(9) |
Oct
(14) |
Nov
(12) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2008 |
Jan
|
Feb
(18) |
Mar
(3) |
Apr
|
May
|
Jun
(2) |
Jul
(3) |
Aug
|
Sep
|
Oct
(17) |
Nov
(13) |
Dec
|
| 2009 |
Jan
(1) |
Feb
(1) |
Mar
(15) |
Apr
(2) |
May
(18) |
Jun
(1) |
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2010 |
Jan
(1) |
Feb
(5) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(61) |
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
(18) |
Apr
|
May
|
Jun
(5) |
Jul
|
Aug
(1) |
Sep
(2) |
Oct
(4) |
Nov
(10) |
Dec
(9) |
| 2012 |
Jan
(10) |
Feb
(23) |
Mar
|
Apr
(5) |
May
(2) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
(4) |
| 2013 |
Jan
(17) |
Feb
(1) |
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <fcr...@us...> - 2012-02-11 06:07:21
|
Revision: 798
http://safekeep.svn.sourceforge.net/safekeep/?rev=798&view=rev
Author: fcrawford
Date: 2012-02-11 06:07:15 +0000 (Sat, 11 Feb 2012)
Log Message:
-----------
Update version for new release
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-02-11 06:05:29 UTC (rev 797)
+++ safekeep/trunk/safekeep 2012-02-11 06:07:15 UTC (rev 798)
@@ -70,7 +70,7 @@
default_bandwidth = {}
PROTOCOL = "1.2"
-VERSION = "1.3.3"
+VERSION = "1.4.0"
VEBOSITY_BY_CLASS = {'DBG': 3, 'INFO': 2, 'WARN': 1, 'ERR': 0}
######################################################################
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-02-11 06:05:35
|
Revision: 797
http://safekeep.svn.sourceforge.net/safekeep/?rev=797&view=rev
Author: fcrawford
Date: 2012-02-11 06:05:29 +0000 (Sat, 11 Feb 2012)
Log Message:
-----------
Prepared for 1.4.0 release
Modified Paths:
--------------
safekeep/trunk/ANNOUNCE
safekeep/trunk/safekeep.spec.in
Modified: safekeep/trunk/ANNOUNCE
===================================================================
--- safekeep/trunk/ANNOUNCE 2012-02-11 05:57:17 UTC (rev 796)
+++ safekeep/trunk/ANNOUNCE 2012-02-11 06:05:29 UTC (rev 797)
@@ -1,34 +1,40 @@
-This is release 1.3.3 of SafeKeep, a centralized and easy to use
+This is release 1.4.0 of SafeKeep, a centralized and easy to use
backup application that combines the best features of a mirror
and an incremental backup.
What's new in this release:
- - More fixes for dealing with LVM snapshots.
- - Better handling of other failure conditions.
- - Updates for Python syntax issues and changes.
- - Allow specification of a sender address in e-mail messages.
- - Better handling of ionice(1).
+ - Add the ability to store script files on either client or server.
+ - Rewritten snapshot creation to remove use of "rbind".
+ - Automatic cleanup on next run after an abort.
+ - Support of LVM tagging for snapshots.
+ - Better handling of messages and tracebacks.
+ - A number of other code cleanups and bug fixes.
+ - Updated minimum Python support to Python 2.3.
-Many thanks to Marco Bozzolan and Harald Nehring for the fixes
-that made this release possible, and to Dimi Paun for his continual
-work with SafeKeep, who is currently having a life away from computers.
+Many thanks to Andres Toomsalu, Ken Bass, Joe Steele and Alexander List
+for the fixes that made this release possible, and to Dimi Paun for his
+continual work with SafeKeep.
Sources and binaries are available from the following locations:
- - RedHat EL/CentOS 3,4,5,6 Fedora 8,9,10,11,12,13,14,15,16:
- http://prdownloads.sourceforge.net/safekeep/safekeep-common-1.3.3-1.noarch.rpm
- http://prdownloads.sourceforge.net/safekeep/safekeep-client-1.3.3-1.noarch.rpm
- http://prdownloads.sourceforge.net/safekeep/safekeep-server-1.3.3-1.noarch.rpm
+ - RedHat EL/CentOS 4,5,6 Fedora 8,9,10,11,12,13,14,15,16:
+ http://prdownloads.sourceforge.net/safekeep/safekeep-common-1.4.0-1.noarch.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-client-1.4.0-1.noarch.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-server-1.4.0-1.noarch.rpm
- - Ubuntu Edgy, Dapper, Breezy, Hardy, Karmic, Lucid, Maverick, and Natty:
- http://prdownloads.sourceforge.net/safekeep/safekeep-common_1.3.3_all.deb
- http://prdownloads.sourceforge.net/safekeep/safekeep-client_1.3.3_all.deb
- http://prdownloads.sourceforge.net/safekeep/safekeep-server_1.3.3_all.deb
+ - Ubuntu Dapper, Breezy, Hardy, Lucid, Maverick, Natty, Oneiric and Precise:
+ http://prdownloads.sourceforge.net/safekeep/safekeep-common_1.4.0_all.deb
+ http://prdownloads.sourceforge.net/safekeep/safekeep-client_1.4.0_all.deb
+ http://prdownloads.sourceforge.net/safekeep/safekeep-server_1.4.0_all.deb
- Source:
- http://prdownloads.sourceforge.net/safekeep/safekeep-1.3.3.tar.gz
- http://prdownloads.sourceforge.net/safekeep/safekeep-1.3.3-1.src.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-1.4.0.tar.gz
+ http://prdownloads.sourceforge.net/safekeep/safekeep-1.4.0-1.src.rpm
+NOTE: The minimum version of Python now supported is Python 2.3. If you
+require support of an older version of Python, then you should select an
+earlier release of Safekeep.
+
To find out more about the project visit on our website:
http://safekeep.sourceforge.net
Modified: safekeep/trunk/safekeep.spec.in
===================================================================
--- safekeep/trunk/safekeep.spec.in 2012-02-11 05:57:17 UTC (rev 796)
+++ safekeep/trunk/safekeep.spec.in 2012-02-11 06:05:29 UTC (rev 797)
@@ -25,7 +25,7 @@
Summary: The SafeKeep backup system (common component)
Group: Applications/System
Requires: rdiff-backup
-Requires: python >= 2.2
+Requires: python >= 2.3
%description common
SafeKeep is a client/server backup system which enhances the
@@ -107,6 +107,15 @@
%doc samples/sample.backup
%changelog
+* Sun Feb 12 2012 Frank Crawford <fr...@cr...> 1.4.0-1
+ - Add the ability to store script files on either client or server.
+ - Rewritten snapshot creation to remove use of "rbind".
+ - Automatic cleanup on next run after an abort.
+ - Support of LVM tagging for snapshots.
+ - Better handling of messages and tracebacks.
+ - A number of other code cleanups and bug fixes.
+ - Updated minimum Python support to Python 2.3.
+
* Sun Nov 20 2011 Frank Crawford <fr...@cr...> 1.3.3-1
- More fixes for dealing with LVM snapshots.
- Better handling of other failure conditions.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-02-11 05:57:23
|
Revision: 796
http://safekeep.svn.sourceforge.net/safekeep/?rev=796&view=rev
Author: fcrawford
Date: 2012-02-11 05:57:17 +0000 (Sat, 11 Feb 2012)
Log Message:
-----------
Updated TODO with latest ideas
Modified Paths:
--------------
safekeep/trunk/TODO
Modified: safekeep/trunk/TODO
===================================================================
--- safekeep/trunk/TODO 2012-02-11 05:38:53 UTC (rev 795)
+++ safekeep/trunk/TODO 2012-02-11 05:57:17 UTC (rev 796)
@@ -3,9 +3,9 @@
* Test the sendmail delivery method
* Add tests db dumps
* Avoid snapshotting snapshots
- * Don't snapshot a device if a snapshot is already present
- * Protect against multiple safekeep instance running at once
- * Do not include fifos and sockets by default
+ * Don't snapshot a device if a snapshot is already present (Partially DONE)
+ * Protect against multiple safekeep instance running at once (DONE)
+ * Do not include fifos and sockets by default (DONE)
Feedback from users:
* Øyvind Skaar <os...@op...>: FreeBSD have snapshot capabilities
@@ -19,9 +19,15 @@
* Decide how big the snapshot size should be automagically
* Fully automatic shapshotting mode
* Configuration inheritance
- * Transactional ops on server side, and automagic recovery from errors
+ * Transactional ops on server side, and automagic recovery from errors (Partially DONE)
* Windows support for the client
* Write SE Linux rules to limit access rights (akin to the httpd ones)
* Run the server in a chroot-jail
* Compare mirroring with a straight rsync(1)
+Future (post 1.4.0):
+ * Server-side scripting
+ * In-line scripts
+ * Automatic compression of DB dumps
+ * Preprocessing of configuration prior to client processing
+ * Multi-threading and concurrent backups
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-02-11 05:39:00
|
Revision: 795
http://safekeep.svn.sourceforge.net/safekeep/?rev=795&view=rev
Author: fcrawford
Date: 2012-02-11 05:38:53 +0000 (Sat, 11 Feb 2012)
Log Message:
-----------
Added note about Python 2.2/2.3 support
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-21 03:10:10 UTC (rev 794)
+++ safekeep/trunk/safekeep 2012-02-11 05:38:53 UTC (rev 795)
@@ -34,8 +34,10 @@
# Python 2.2 compatibility
######################################################################
# There is no guarantee that we'll continue supporting Python 2.2
-# indefinitely, but we make a reasonable effor to do so as long as
+# indefinitely, but we make a reasonable effort to do so as long as
# it doesn't result in major complication/ugliness.
+# Note that some features related to scripting, snapshots and DB backups
+# already require Python 2.3.
try:
True, False
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-21 03:10:16
|
Revision: 794
http://safekeep.svn.sourceforge.net/safekeep/?rev=794&view=rev
Author: fcrawford
Date: 2012-01-21 03:10:10 +0000 (Sat, 21 Jan 2012)
Log Message:
-----------
Tie message to specific backup configuration
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-20 13:21:54 UTC (rev 793)
+++ safekeep/trunk/safekeep 2012-01-21 03:10:10 UTC (rev 794)
@@ -334,7 +334,7 @@
if tag:
if not tag.startswith('@'): tag = '@' + tag.lstrip()
tags.append(tag.strip())
- elif not is_client:
+ elif is_client:
warn('Device: %s: empty tag in taglist: %s' % (device, tag_el))
return { 'device' : device, 'size' : size, 'tags' : tags }
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-20 13:22:05
|
Revision: 793
http://safekeep.svn.sourceforge.net/safekeep/?rev=793&view=rev
Author: fcrawford
Date: 2012-01-20 13:21:54 +0000 (Fri, 20 Jan 2012)
Log Message:
-----------
Suppress duplicate message from client
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-20 12:25:15 UTC (rev 792)
+++ safekeep/trunk/safekeep 2012-01-20 13:21:54 UTC (rev 793)
@@ -334,7 +334,7 @@
if tag:
if not tag.startswith('@'): tag = '@' + tag.lstrip()
tags.append(tag.strip())
- else:
+ elif not is_client:
warn('Device: %s: empty tag in taglist: %s' % (device, tag_el))
return { 'device' : device, 'size' : size, 'tags' : tags }
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-20 12:25:21
|
Revision: 792
http://safekeep.svn.sourceforge.net/safekeep/?rev=792&view=rev
Author: fcrawford
Date: 2012-01-20 12:25:15 +0000 (Fri, 20 Jan 2012)
Log Message:
-----------
Allow snapshot tag item to take a comma separated list of tags
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/safekeep
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2012-01-13 11:47:11 UTC (rev 791)
+++ safekeep/trunk/doc/safekeep.backup.txt 2012-01-20 12:25:15 UTC (rev 792)
@@ -249,9 +249,10 @@
Mandatory for a `<snapshot>` element.
/backup/setup/snapshot/@tag::
- A tag to be added to the snapshot, with the `--addtag` argument
- to `lvcreate`. An `@` is automatically added to the generated
- tag.
+ A list of tags to be added to the snapshot, with the `--addtag`
+ argument to `lvcreate`. The @tag entry consists of a `,`
+ separated list of tags.
+ An `@` is automatically added to each generated tag.
Optional for a `<snapshot>` element.
/backup/setup/script/@location::
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-13 11:47:11 UTC (rev 791)
+++ safekeep/trunk/safekeep 2012-01-20 12:25:15 UTC (rev 792)
@@ -327,8 +327,16 @@
size = snap_el.getAttribute('size')
if not size:
raise ConfigException('Please specify the size for the snapshot')
- tag = snap_el.getAttribute('tag')
- return { 'device' : device, 'size' : size, 'tag' : tag }
+ tags = []
+ tag_el = snap_el.getAttribute('tag')
+ if tag_el:
+ for tag in tag_el.split(','):
+ if tag:
+ if not tag.startswith('@'): tag = '@' + tag.lstrip()
+ tags.append(tag.strip())
+ else:
+ warn('Device: %s: empty tag in taglist: %s' % (device, tag_el))
+ return { 'device' : device, 'size' : size, 'tags' : tags }
def parse_clude(clude_el):
path = clude_el.getAttribute('path')
@@ -690,14 +698,12 @@
if not snapmnt:
warn('Cannot find the mountpoint for: %s' % device)
continue
- tag = snap['tag']
- if tag:
- args = ['lvcreate', '--addtag', '@' + tag,
- '--snapshot', '--size', snap['size'],
- '--name', os.path.basename(snapdev), lvmdev]
- else:
- args = ['lvcreate', '--snapshot', '--size', snap['size'],
- '--name', os.path.basename(snapdev), lvmdev]
+ args = ['lvcreate']
+ if snap['tags']:
+ for tag in snap['tags']:
+ args.extend(['--addtag', tag.strip()])
+ args.extend(['--snapshot', '--size', snap['size'],
+ '--name', os.path.basename(snapdev), lvmdev])
ec = spawn(args)
if ec:
warn('Can not snapshot the device: %s' % device)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-13 11:47:22
|
Revision: 791
http://safekeep.svn.sourceforge.net/safekeep/?rev=791&view=rev
Author: fcrawford
Date: 2012-01-13 11:47:11 +0000 (Fri, 13 Jan 2012)
Log Message:
-----------
Clean up simple coding issues reported by pylint
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-08 00:01:59 UTC (rev 790)
+++ safekeep/trunk/safekeep 2012-01-13 11:47:11 UTC (rev 791)
@@ -66,7 +66,6 @@
base_dir = None
current_pid = os.getpid()
default_bandwidth = {}
-cmd = "<Missing>"
PROTOCOL = "1.2"
VERSION = "1.3.3"
@@ -110,10 +109,10 @@
else:
print >> sys.stderr, msg.encode('utf-8')
-def info_file(file, marker=None):
- info('# File: ' + file)
- errs = 0;
- fin = open(file, 'r')
+def info_file(filename, marker=None):
+ info('# File: ' + filename)
+ errs = 0
+ fin = open(filename, 'r')
try:
for line in fin.readlines():
if marker:
@@ -183,7 +182,7 @@
child_in.write(stdin)
child_in.close()
- lines=[]
+ lines = []
for line in child_out:
if stdout:
lines.append(line)
@@ -207,7 +206,7 @@
rc, out = do_spawn(args, stdin, stdout)
except OSError, ex:
ret = "OSError: %s" % (ex)
- error('%s failed: %s' % (cmd, ret));
+ error('%s failed: %s' % (cmd, ret))
return ret
if not rc:
@@ -219,7 +218,7 @@
else:
ret = 'unknown exit status: %d' % rc
if ret:
- error('%s failed: %s' % (cmd, ret));
+ error('%s failed: %s' % (cmd, ret))
return (ret, out)
# this just spawns an external program (optionally through a shell)
@@ -242,7 +241,7 @@
rc, out = do_spawn(args, None, True)
except OSError, ex:
return None
- if not rc in (0,1):
+ if not rc in (0, 1):
return None
return out or ''
@@ -265,8 +264,8 @@
cmd = ['/usr/sbin/sendmail', '-t', '-f', email_from]
call(cmd, stdin=msg)
-def is_temp_root(dir):
- return dir != '/'
+def is_temp_root(directory):
+ return directory != '/'
def reroot(root, path):
if root == '/': return path
@@ -275,9 +274,9 @@
if path.startswith('/'): return root + path
return os.path.join(root, path)
-def parse_prop_file(file):
+def parse_prop_file(filename):
props = {}
- fin = open(file)
+ fin = open(filename)
lines = fin.readlines()
fin.close()
for line in lines:
@@ -301,23 +300,23 @@
return repr(self.value)
def parse_dump(dump_el):
- type = dump_el.getAttribute('type')
- if not type:
+ dbtype = dump_el.getAttribute('type')
+ if not dbtype:
raise ConfigException('You need to specify the database type')
- if type not in ('postgres', 'postgresql', 'pgsql', 'mysql'):
- raise ConfigException('Invalid database type: %s' % type)
+ if dbtype not in ('postgres', 'postgresql', 'pgsql', 'mysql'):
+ raise ConfigException('Invalid database type: %s' % dbtype)
db = dump_el.getAttribute('db')
user = dump_el.getAttribute('user')
dbuser = dump_el.getAttribute('dbuser')
dbpasswd = dump_el.getAttribute('dbpasswd')
opts = (dump_el.getAttribute('options') or '').split()
- file = dump_el.getAttribute('file')
- if not file:
+ dbfile = dump_el.getAttribute('file')
+ if not dbfile:
raise ConfigException('You need to specify where the database should be dumped')
cleanup = dump_el.getAttribute('cleanup')
- return { 'type' : type, 'db' : db, 'user' : user, 'dbuser' : dbuser, 'dbpasswd': dbpasswd,
- 'opts' : opts, 'file' : file, 'cleanup' : cleanup }
+ return { 'type' : dbtype, 'db' : db, 'user' : user, 'dbuser' : dbuser, 'dbpasswd': dbpasswd,
+ 'opts' : opts, 'file' : dbfile, 'cleanup' : cleanup }
def parse_snap(snap_el):
device = snap_el.getAttribute('device')
@@ -358,8 +357,8 @@
def parse_config(backup_el, dflt_id):
if backup_el.tagName != 'backup':
raise ConfigException('Invalid config file, the top level element must be <backup>')
- id = backup_el.getAttribute('id')
- if not id: id = dflt_id
+ cfg_id = backup_el.getAttribute('id')
+ if not cfg_id: cfg_id = dflt_id
host_el = backup_el.getElementsByTagName('host')
if host_el:
@@ -392,15 +391,15 @@
raise ConfigException('Can not have more than one bandwidth element')
repo_el = backup_el.getElementsByTagName('repo')
- dir = None
+ repo_dir = None
retention = None
if len(repo_el) == 1:
- dir = repo_el[0].getAttribute('path')
+ repo_dir = repo_el[0].getAttribute('path')
retention = repo_el[0].getAttribute('retention')
elif len(repo_el) > 1:
raise ConfigException('Can not have more than one repo element')
- if not dir: dir = id
- dir = os.path.join(base_dir, dir)
+ if not repo_dir: repo_dir = cfg_id
+ repo_dir = os.path.join(base_dir, repo_dir)
options_els = backup_el.getElementsByTagName('options')
options = []
@@ -410,7 +409,7 @@
continue
option = options_el.nodeName
if option == 'special-files':
- warn('options element special-files is deprecated, use data attributes instead')
+ warn('options element special-files is deprecated, use data attributes instead')
if option in ('special-files', 'rdiff-backup'):
if options_el.hasAttributes():
for key, value in options_el.attributes.items():
@@ -466,8 +465,8 @@
'/var/named/chroot/var/run', '/var/named/chroot/var/tmp' ]
cludes = [{ 'type' : 'exclude', 'path' : path, 'glob' : None, 'regexp' : None } for path in path_xcludes]
- return { 'id': id, 'host' : host, 'port' : port, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
- 'dir' : dir, 'retention' : retention, 'dumps' : dumps, 'snaps' : snaps, 'script' : script,
+ return { 'id': cfg_id, 'host' : host, 'port' : port, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
+ 'dir' : repo_dir, 'retention' : retention, 'dumps' : dumps, 'snaps' : snaps, 'script' : script,
'cludes' : cludes, 'data_options' : data_options, 'options' : options, 'bw' : bw}
def parse_locs(cfglocs):
@@ -548,10 +547,10 @@
def do_client_dbdump(cfg):
debug('Doing DB dumps')
for dump in cfg['dumps']:
- type = dump['type']
+ dbtype = dump['type']
opts = dump['opts']
passwdfile = None
- if type in ('postgres', 'postgresql', 'pgsql'):
+ if dbtype in ('postgres', 'postgresql', 'pgsql'):
if dump['db']:
args = ['pg_dump']
args.extend(['-C'])
@@ -568,7 +567,7 @@
f.write(dump['dbpasswd'])
f.close()
- elif type in ('mysql'):
+ elif dbtype in ('mysql'):
args = ['mysqldump']
if dump['dbuser']:
args.extend(['-u', dump['dbuser']])
@@ -581,7 +580,7 @@
args.extend([dump['db']])
else:
- warn('Invalid database type: ' + type)
+ warn('Invalid database type: ' + dbtype)
continue
if dump['user']:
@@ -592,7 +591,7 @@
if passwdfile:
- os.environ['PGPASSFILE'] = passwdfile
+ os.environ['PGPASSFILE'] = passwdfile
try:
ec = spawn(cmd)
finally:
@@ -656,11 +655,11 @@
def do_lvremove(device):
(group, volume) = device.split('/')[-2:]
if group == 'mapper':
- lvmdev = device;
+ lvmdev = device
else:
lvmdev = '/dev/mapper/%s-%s' % (group, volume.replace('-', '--'))
if os.path.exists(lvmdev):
- for i in range(1,10):
+ for i in range(1, 10):
ret = spawn(['sync'])
ret = spawn(['dmsetup', 'remove', lvmdev])
ret = spawn(['dmsetup', 'remove', lvmdev + '-cow'])
@@ -1025,9 +1024,9 @@
client_side_script('POST-SETUP', cfg, bdir)
send('OK ' + bdir)
elif line.startswith('CLEANUP'):
- dir = line[7:].strip()
- if dir == bdir: should_cleanup = False
- do_client_cleanup(cfg, dir)
+ path = line[7:].strip()
+ if path == bdir: should_cleanup = False
+ do_client_cleanup(cfg, path)
client_side_script('POST-BACKUP', cfg, bdir)
send('OK')
elif line.startswith('SCRUB'):
@@ -1113,15 +1112,15 @@
limit_ul = get_bandwidth(cfg, 'upload')
if limit_dl or limit_ul:
trickle.extend([trickle_cmd])
- if verbosity_trickle: trickle.extend([verbosity_trickle])
+ if verbosity_trickle: trickle.extend([verbosity_trickle])
if limit_dl:
trickle.extend(['-d', str(limit_dl)])
if limit_ul:
trickle.extend(['-u', str(limit_ul)])
if len(trickle):
if try_to_run([trickle_cmd, '-V']) is None:
- warn('Trickle not available, bandwidth limiting disabled')
- trickle = []
+ warn('Trickle not available, bandwidth limiting disabled')
+ trickle = []
args.extend(trickle)
args.extend(['rdiff-backup'])
@@ -1139,7 +1138,7 @@
special_files = []
if cfg['data_options'].get('exclude-devices').lower() == 'true':
- special_files.extend(['--exclude-device-files'])
+ special_files.extend(['--exclude-device-files'])
if cfg['data_options'].get('exclude-sockets').lower() == 'true':
special_files.extend(['--exclude-sockets'])
if cfg['data_options'].get('exclude-fifos').lower() == 'true':
@@ -1203,10 +1202,10 @@
debug("Do server main loop")
output_done = False
for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
+ cfg_id = cfg['id']
+ if ids and cfg_id not in ids: continue
info('------------------------------------------------------------------')
- info('Server backup starting for client %s' % id)
+ info('Server backup starting for client %s' % cfg_id)
output_done = True
cleaned_up = True
@@ -1231,9 +1230,9 @@
cmd = []
if cfg['host']:
cmd.extend(['ssh'])
- if verbosity_ssh: cmd.extend([verbosity_ssh])
+ if verbosity_ssh: cmd.extend([verbosity_ssh])
if cfg['port']: cmd.extend(['-p', cfg['port']])
- cmd.extend(['-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host']])
+ cmd.extend(['-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host']])
cmd.extend(['safekeep', '--client'])
if use_subprocess:
@@ -1248,7 +1247,7 @@
client_versions = do_server_getanswer(cout)
do_server_compat(client_versions)
- cin.write('CONFIG: %d: %s\n' % (len(cfg['text'].splitlines()), id))
+ cin.write('CONFIG: %d: %s\n' % (len(cfg['text'].splitlines()), cfg_id))
cin.write(cfg['text'] + '\n')
cin.flush()
remote_script = do_server_getanswer(cout)
@@ -1316,19 +1315,19 @@
do_server_getanswer(cout)
if errs == 0:
- info('Server backup for client %s: OK' % id)
+ info('Server backup for client %s: OK' % cfg_id)
else:
- info('Server backup for client %s: OK (%d WARNINGS)' % (id, errs))
+ info('Server backup for client %s: OK (%d WARNINGS)' % (cfg_id, errs))
except Exception, ex:
if cleanup:
- info('Client-side cleanup for client %s: FAILED' % id)
+ info('Client-side cleanup for client %s: FAILED' % cfg_id)
else:
- if isinstance(ex, ClientException):
- error('Client %s: FAILED due to: %s' % (id, ex or ''))
- if ex.traceback and verbosity_level > 2: error(ex.traceback)
- else:
- error('Server backup for client %s: FAILED' % id, ex)
+ if isinstance(ex, ClientException):
+ error('Client %s: FAILED due to: %s' % (cfg_id, ex or ''))
+ if ex.traceback and verbosity_level > 2: error(ex.traceback)
+ else:
+ error('Server backup for client %s: FAILED' % cfg_id, ex)
# Shutdown client
cout.close()
@@ -1346,13 +1345,13 @@
debug("Do server listing main loop")
output_done = False
for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
+ cfg_id = cfg['id']
+ if ids and cfg_id not in ids: continue
if list_parsable:
- info('Client: %s' % id)
+ info('Client: %s' % cfg_id)
else:
info('------------------------------------------------------------------')
- info('Server listing for client %s' % id)
+ info('Server listing for client %s' % cfg_id)
output_done = True
args = ['rdiff-backup']
@@ -1382,11 +1381,11 @@
def do_keys(cfgs, ids, nice_rem, identity, status, dump, deploy):
for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
- info('Handling keys for client: %s' % id)
+ cfg_id = cfg['id']
+ if ids and cfg_id not in ids: continue
+ info('Handling keys for client: %s' % cfg_id)
if not cfg['host']:
- info('%s: Client is local, it needs no keys' % id)
+ info('%s: Client is local, it needs no keys' % cfg_id)
continue
nice = cfg['nice'] or nice_rem
@@ -1403,25 +1402,25 @@
publickeyfile = privatekeyfile + '.pub'
if not os.path.isfile(privatekeyfile):
if os.path.isfile(publickeyfile):
- error('%s: Public key exists %s, but private key is missing. Skipping client.' % (id, publickeyfile))
+ error('%s: Public key exists %s, but private key is missing. Skipping client.' % (cfg_id, publickeyfile))
break
if dump:
- print '%s: Key does not exist: %s.' % (id, privatekeyfile)
+ print '%s: Key does not exist: %s.' % (cfg_id, privatekeyfile)
break
if status:
- print '%s: Key does not exist: %s. Will be generated.' % (id, privatekeyfile)
+ print '%s: Key does not exist: %s. Will be generated.' % (cfg_id, privatekeyfile)
break
if deploy:
- info('%s: Key do not exist, generating it now: %s' % (id, privatekeyfile))
+ info('%s: Key do not exist, generating it now: %s' % (cfg_id, privatekeyfile))
gencmd = 'ssh-keygen -q -b 1024 -t dsa -N "" -C "SafeKeep auto generated key at %s@%s" -f %s' % (backup_user, os.uname()[1], privatekeyfile)
if backup_user != work_user:
gencmd = 'su -s /bin/sh -c %s - %s' % (commands.mkarg(gencmd), backup_user)
debug(gencmd)
if spawn(gencmd):
- error('%s: Failed to generate key %s. Skipping client.' % (id, privatekeyfile))
+ error('%s: Failed to generate key %s. Skipping client.' % (cfg_id, privatekeyfile))
break
if not os.path.isfile(publickeyfile):
- error('%s: Private key exists %s, but public key is missing. Skipping client.' % (id, privatekeyfile))
+ error('%s: Private key exists %s, but public key is missing. Skipping client.' % (cfg_id, privatekeyfile))
break
fin = open(publickeyfile, 'r')
publickey = fin.read()
@@ -1445,7 +1444,7 @@
cmd = [basessh, '%s@%s' % (cfg['user'], cfg['host']), "if test -f .ssh/authorized_keys; then cat .ssh/authorized_keys; fi"]
authtext = call(cmd)
if authtext is None:
- error('%s: Failed to read the %s@%s:~/.ssh/authorized_keys file.' % (id, cfg['user'], cfg['host']))
+ error('%s: Failed to read the %s@%s:~/.ssh/authorized_keys file.' % (cfg_id, cfg['user'], cfg['host']))
continue
auth_keys = parse_authorized_keys(authtext)
this_keys = parse_authorized_keys(output)
@@ -1457,11 +1456,11 @@
new_keys.append(this_key)
if not new_keys:
if status:
- print '%s: Client is up to date.' % id
+ print '%s: Client is up to date.' % cfg_id
continue
if status:
- print '%s: Keys will be deployed on the client.' % id
+ print '%s: Keys will be deployed on the client.' % cfg_id
if deploy:
cmd = [basessh, '%s@%s' % (cfg['user'], cfg['host']), "umask 077; test -d .ssh || mkdir .ssh; cat >> .ssh/authorized_keys"]
keys = '%s\n' % '\n'.join([key[4] for key in new_keys])
@@ -1479,7 +1478,7 @@
if line[0] in '0123456789':
warn('SSH Protocol 1 keys are ignored: %s' % line)
continue
- opts =''
+ opts = ''
if line[0:7] not in ('ssh-dss', 'ssh-rsa'):
in_str = False
in_esc = False
@@ -1509,9 +1508,9 @@
error('Invalid key line, skipping: %s' % line)
continue
- type = parts[0]
- if type not in ('ssh-dss', 'ssh-rsa'):
- error('Invalid key type "%s", skipping: %s' % (type, line))
+ ssh_type = parts[0]
+ if ssh_type not in ('ssh-dss', 'ssh-rsa'):
+ error('Invalid key type "%s", skipping: %s' % (ssh_type, line))
continue
base46enc = parts[1]
@@ -1521,7 +1520,7 @@
else:
comment = parts[2]
- keys.append((opts, type, base46enc, comment, line))
+ keys.append((opts, ssh_type, base46enc, comment, line))
return keys
@@ -1697,7 +1696,7 @@
if cfgfile:
warn('Configuration file does not exist, skipping: %s' % cfgfile)
else:
- cfgfile = config_file
+ cfgfile = config_file
props = {}
def get_int(p):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-08 00:02:05
|
Revision: 790
http://safekeep.svn.sourceforge.net/safekeep/?rev=790&view=rev
Author: fcrawford
Date: 2012-01-08 00:01:59 +0000 (Sun, 08 Jan 2012)
Log Message:
-----------
Correctly suppress any error message
Modified Paths:
--------------
safekeep/trunk/safekeep.cron
Modified: safekeep/trunk/safekeep.cron
===================================================================
--- safekeep/trunk/safekeep.cron 2012-01-02 03:17:45 UTC (rev 789)
+++ safekeep/trunk/safekeep.cron 2012-01-08 00:01:59 UTC (rev 790)
@@ -2,6 +2,6 @@
PATH=/sbin:/bin:/usr/sbin:/usr/bin
-if ls /etc/safekeep/backup.d/*.backup 2>&1 1>/dev/null; then
+if ls /etc/safekeep/backup.d/*.backup 1>/dev/null 2>&1; then
safekeep -v --server
fi
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-02 03:17:51
|
Revision: 789
http://safekeep.svn.sourceforge.net/safekeep/?rev=789&view=rev
Author: fcrawford
Date: 2012-01-02 03:17:45 +0000 (Mon, 02 Jan 2012)
Log Message:
-----------
Corrected test for usable trickle as reported by Ken Bass
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-01 07:59:55 UTC (rev 788)
+++ safekeep/trunk/safekeep 2012-01-02 03:17:45 UTC (rev 789)
@@ -1119,7 +1119,7 @@
if limit_ul:
trickle.extend(['-u', str(limit_ul)])
if len(trickle):
- if try_to_run([trickle_cmd, '-V']) is not None:
+ if try_to_run([trickle_cmd, '-V']) is None:
warn('Trickle not available, bandwidth limiting disabled')
trickle = []
args.extend(trickle)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-01 08:00:01
|
Revision: 788
http://safekeep.svn.sourceforge.net/safekeep/?rev=788&view=rev
Author: fcrawford
Date: 2012-01-01 07:59:55 +0000 (Sun, 01 Jan 2012)
Log Message:
-----------
Improved coding for script location, including change of default location
to be client first and then server, test of permissions on server based
script and notification of errors and missing file on the server side.
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/safekeep
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2012-01-01 04:58:17 UTC (rev 787)
+++ safekeep/trunk/doc/safekeep.backup.txt 2012-01-01 07:59:55 UTC (rev 788)
@@ -264,8 +264,8 @@
The `location` optionally consists of an optional `host` and
a mandatory `path`, separated by a ":", where the host part is
either "client" or "server". If no host part is specified then
- it is first looked for on the server, and if not found, then is
- looked for on the client. If it not found on either, then a
+ it is first looked for on the client, and if not found, then is
+ looked for on the server. If it not found on either, then a
warning is raised.
See the `CLIENT SCRIPT` section for more information.
Mandatory for a `<script>` element.
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-01 04:58:17 UTC (rev 787)
+++ safekeep/trunk/safekeep 2012-01-01 07:59:55 UTC (rev 788)
@@ -434,8 +434,12 @@
script_el = setup_el[0].getElementsByTagName('script')
if len(script_el) == 1:
script = script_el[0].getAttribute('path')
- if not ':' in script and os.path.isfile(script):
- script = 'server:' + script
+ if not ':' in script:
+ if is_client and os.path.isfile(script):
+ warn('Assuming client based script: %s' % script)
+ script = 'client:' + script
+ else:
+ script = 'server:' + script
elif len(script_el) > 1:
raise ConfigException('Can not have more than one setup script element')
@@ -779,15 +783,22 @@
def do_client_remote_script(script_file, cfg, cmd):
(cfg_cmd, server_file, cnt_str) = cmd.split(':', 2)
debug("do_client_remote_script: %s -> %s: cnt_str = %s" % (server_file, cfg['script'], cnt_str.strip()))
- try:
- for i in xrange(int(cnt_str)):
- line = sys.stdin.readline()
- if not line: raise Exception('Unexpected end of file')
- script_file.write(line)
- finally:
+ if int(cnt_str) > 0:
+ try:
+ for i in xrange(int(cnt_str)):
+ line = sys.stdin.readline()
+ if not line: raise Exception('Unexpected end of file')
+ script_file.write(line)
+ finally:
+ script_file.close()
+
+ os.chmod(cfg['script'], stat.S_IXUSR | stat.S_IRUSR)
+ return True
+ else:
script_file.close()
+ os.remove(cfg['script'])
- os.chmod(cfg['script'], stat.S_IXUSR | stat.S_IRUSR)
+ return False
def do_client_setup(cfg):
debug('Do setup of %s' % cfg['host'])
@@ -999,11 +1010,14 @@
warn('Unknown script location %s for script %s' % (script_loc, script))
send('OK')
elif line.startswith('SCRIPT'):
- do_client_remote_script(script_file, cfg, line)
- ret = client_side_script('STARTUP', cfg, bdir)
- if ret:
- send('ERROR Client-side setup script failed: %s' % ret)
+ if do_client_remote_script(script_file, cfg, line):
+ ret = client_side_script('STARTUP', cfg, bdir)
+ if ret:
+ send('ERROR Client-side setup script failed: %s' % ret)
+ else:
+ send('OK')
else:
+ cfg['script'] = None
send('OK')
elif line.startswith('SETUP'):
client_side_script('PRE-SETUP', cfg, bdir)
@@ -1240,12 +1254,21 @@
remote_script = do_server_getanswer(cout)
if cfg['script'].startswith('server:') and remote_script:
local_script = cfg['script'].split(':', 1)[1]
- debug("Transferring script: %s -> %s" % (local_script, remote_script))
- fscript = open(local_script)
- lines = fscript.readlines()
- fscript.close()
- cin.write('SCRIPT: %s: %d\n' % (local_script, len(lines)))
- cin.writelines(lines)
+ if os.path.isfile(local_script):
+ ret = check_script_permissions(local_script)
+ if not ret:
+ debug("Transferring script: %s -> %s" % (local_script, remote_script))
+ fscript = open(local_script)
+ lines = fscript.readlines()
+ fscript.close()
+ cin.write('SCRIPT: %s: %d\n' % (local_script, len(lines)))
+ cin.writelines(lines)
+ else:
+ error('Illegal script specified: %s: %s' % (local_script, ret))
+ cin.write('SCRIPT: %s: %d\n' % ('-', 0))
+ else:
+ warn('No server based script found: %s' % local_script)
+ cin.write('SCRIPT: %s: %d\n' % ('-', 0))
cin.flush()
do_server_getanswer(cout)
if cleanup:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-01 04:58:24
|
Revision: 787
http://safekeep.svn.sourceforge.net/safekeep/?rev=787&view=rev
Author: fcrawford
Date: 2012-01-01 04:58:17 +0000 (Sun, 01 Jan 2012)
Log Message:
-----------
Fix up verbosity_level to allow debug messages within configuration parsing
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-01 03:31:10 UTC (rev 786)
+++ safekeep/trunk/safekeep 2012-01-01 04:58:17 UTC (rev 787)
@@ -1556,6 +1556,8 @@
usage(2)
global backup_user, home_dir, base_dir
+ global verbosity_level
+
mode = None
email = []
email_from = None
@@ -1726,6 +1728,7 @@
if len(cfglocs) > 0:
try:
+ verbosity_level = 1 + verbosity
cfgs = parse_locs(cfglocs)
except Exception, ex:
if isinstance(ex, ConfigException):
@@ -1749,7 +1752,7 @@
if not ok: sys.exit(2)
try:
- global is_client, verbosity_level, verbosity_ssh, verbosity_trickle
+ global is_client, verbosity_ssh, verbosity_trickle
if verbosity > 2:
verbosity_trickle = verbosity_ssh = '-' + (verbosity-2) * 'v'
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-01 03:31:16
|
Revision: 786
http://safekeep.svn.sourceforge.net/safekeep/?rev=786&view=rev
Author: fcrawford
Date: 2012-01-01 03:31:10 +0000 (Sun, 01 Jan 2012)
Log Message:
-----------
Better reporting of server configuration errors, similar to other exception handling
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2012-01-01 03:28:15 UTC (rev 785)
+++ safekeep/trunk/safekeep 2012-01-01 03:31:10 UTC (rev 786)
@@ -1725,7 +1725,14 @@
base_dir = home_dir
if len(cfglocs) > 0:
- cfgs = parse_locs(cfglocs)
+ try:
+ cfgs = parse_locs(cfglocs)
+ except Exception, ex:
+ if isinstance(ex, ConfigException):
+ error('CONFIG ERROR: %s' % (ex or ''), ex)
+ else:
+ error('ERROR: %s' % (ex or ''), ex)
+ sys.exit(2)
else:
cfgs = {}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2012-01-01 03:28:22
|
Revision: 785
http://safekeep.svn.sourceforge.net/safekeep/?rev=785&view=rev
Author: fcrawford
Date: 2012-01-01 03:28:15 +0000 (Sun, 01 Jan 2012)
Log Message:
-----------
Added option to set SSH port for each host as requested by Ken Bass
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/safekeep
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2011-12-30 09:27:22 UTC (rev 784)
+++ safekeep/trunk/doc/safekeep.backup.txt 2012-01-01 03:28:15 UTC (rev 785)
@@ -88,6 +88,12 @@
This is different from leaving it blank, as it will establish a
SSH session and use the user specified by `/backup/host/@user`.
Optional, defaults to local access.
+
+/backup/host/@port::
+ The network port to use when connecting to the client. This must
+ be a number and is passed to the network connection agent, usually
+ SSH.
+ Optional, default to not passing any port specification.
/backup/host/@user::
The user name to use when connecting to the client. This user
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-30 09:27:22 UTC (rev 784)
+++ safekeep/trunk/safekeep 2012-01-01 03:28:15 UTC (rev 785)
@@ -364,12 +364,15 @@
host_el = backup_el.getElementsByTagName('host')
if host_el:
host = host_el[0].getAttribute('name')
+ port = host_el[0].getAttribute('port')
user = host_el[0].getAttribute('user')
nice = host_el[0].getAttribute('nice')
key_ctrl = host_el[0].getAttribute('key-ctrl')
key_data = host_el[0].getAttribute('key-data')
else:
- host = user = nice = key_ctrl = key_data = None
+ host = port = user = nice = key_ctrl = key_data = None
+ if host and port and not port.isdigit():
+ raise ConfigException('Host port must be a number: "%s"' % port)
if host and not user:
user = 'root'
if host and not key_ctrl:
@@ -459,7 +462,7 @@
'/var/named/chroot/var/run', '/var/named/chroot/var/tmp' ]
cludes = [{ 'type' : 'exclude', 'path' : path, 'glob' : None, 'regexp' : None } for path in path_xcludes]
- return { 'id': id, 'host' : host, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
+ return { 'id': id, 'host' : host, 'port' : port, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
'dir' : dir, 'retention' : retention, 'dumps' : dumps, 'snaps' : snaps, 'script' : script,
'cludes' : cludes, 'data_options' : data_options, 'options' : options, 'bw' : bw}
@@ -1110,7 +1113,9 @@
args.extend(['rdiff-backup'])
if cfg['host']:
- schema = 'ssh %s -i %s %%s rdiff-backup --server' % (verbosity_ssh, cfg['key_data'])
+ basessh = 'ssh'
+ if cfg['port']: basessh += ' -p %s' % cfg['port']
+ schema = '%s %s -i %s %%s rdiff-backup --server' % (basessh, verbosity_ssh, cfg['key_data'])
args.extend(['--remote-schema', schema])
if force:
@@ -1213,6 +1218,7 @@
if cfg['host']:
cmd.extend(['ssh'])
if verbosity_ssh: cmd.extend([verbosity_ssh])
+ if cfg['port']: cmd.extend(['-p', cfg['port']])
cmd.extend(['-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host']])
cmd.extend(['safekeep', '--client'])
@@ -1409,6 +1415,7 @@
print output
basessh = 'ssh'
+ if cfg['port']: basessh += ' -p %s' % cfg['port']
if identity: basessh += ' -i %s' % (commands.mkarg(identity))
if status or deploy:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-30 09:27:28
|
Revision: 784
http://safekeep.svn.sourceforge.net/safekeep/?rev=784&view=rev
Author: fcrawford
Date: 2011-12-30 09:27:22 +0000 (Fri, 30 Dec 2011)
Log Message:
-----------
Add the ability to have the script file on either the client or the
server. Note that it is still only run on the client, just that for
a server located one, a temporary copy is sent to the client to run.
At present, it is designed around passing scripts, not binaries, as it
passes the file line by line.
The protocol version has been bumped to 1.2, due to an addition to rund
the script after it is copied.
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/safekeep
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2011-12-30 08:50:25 UTC (rev 783)
+++ safekeep/trunk/doc/safekeep.backup.txt 2011-12-30 09:27:22 UTC (rev 784)
@@ -47,7 +47,7 @@
<snapshot device="/path/to/volume" size="500M" />
<!-- location of a script to be executed at different stages of the run -->
- <script path="/path/to/script" />
+ <script path="server:/path/to/script" />
</setup>
@@ -248,15 +248,21 @@
tag.
Optional for a `<snapshot>` element.
-/backup/setup/script/@path::
- Execute the script specified path on the client at certain steps
- of the backup process.
- This script is executed with three arguments:
- - Backup id (/backup/@id)
- - Backup step
- - Backup root directory (valid after creation of a snapshot)
- See the `CLIENT SCRIPT` section for more information.
- Mandatory for a `<script>` element.
+/backup/setup/script/@location::
+ Execute the script specified path on the client at certain steps
+ of the backup process.
+ This script is executed with three arguments:
+ - Backup id (/backup/@id)
+ - Backup step
+ - Backup root directory (valid after creation of a snapshot)
+ The `location` optionally consists of an optional `host` and
+ a mandatory `path`, separated by a ":", where the host part is
+ either "client" or "server". If no host part is specified then
+ it is first looked for on the server, and if not found, then is
+ looked for on the client. If it not found on either, then a
+ warning is raised.
+ See the `CLIENT SCRIPT` section for more information.
+ Mandatory for a `<script>` element.
/backup/data/@exclude-devices::
One of "true" or "false". If "true", the dump file will
@@ -343,6 +349,15 @@
`safekeep(1)` support the optional execution of a script or program
on the client system at different steps during execution of the backup.
+The script may be located on either the server or the client. If it is
+located on the server, then the file is copied to the client into a
+temporary directory before execution. Note that this directory is located
+where ever the system normally creates temporary files, and it is possible
+that the execution of scripts are disallowed. In that case it is recommended
+that a client based script is used. In addition, the script is copied from
+the server on a line by line basis, and so it is not suitable to pass binary
+files.
+
Note: specification of a script which does not exist is not considered an
error, and is treated as the same as not specifying a script. However, if
the specified path does match a file or directory, the security tests listed
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-30 08:50:25 UTC (rev 783)
+++ safekeep/trunk/safekeep 2011-12-30 09:27:22 UTC (rev 784)
@@ -68,7 +68,7 @@
default_bandwidth = {}
cmd = "<Missing>"
-PROTOCOL = "1.1"
+PROTOCOL = "1.2"
VERSION = "1.3.3"
VEBOSITY_BY_CLASS = {'DBG': 3, 'INFO': 2, 'WARN': 1, 'ERR': 0}
@@ -431,6 +431,8 @@
script_el = setup_el[0].getElementsByTagName('script')
if len(script_el) == 1:
script = script_el[0].getAttribute('path')
+ if not ':' in script and os.path.isfile(script):
+ script = 'server:' + script
elif len(script_el) > 1:
raise ConfigException('Can not have more than one setup script element')
@@ -771,6 +773,19 @@
finally:
dom.unlink()
+def do_client_remote_script(script_file, cfg, cmd):
+ (cfg_cmd, server_file, cnt_str) = cmd.split(':', 2)
+ debug("do_client_remote_script: %s -> %s: cnt_str = %s" % (server_file, cfg['script'], cnt_str.strip()))
+ try:
+ for i in xrange(int(cnt_str)):
+ line = sys.stdin.readline()
+ if not line: raise Exception('Unexpected end of file')
+ script_file.write(line)
+ finally:
+ script_file.close()
+
+ os.chmod(cfg['script'], stat.S_IXUSR | stat.S_IRUSR)
+
def do_client_setup(cfg):
debug('Do setup of %s' % cfg['host'])
@@ -892,7 +907,7 @@
warn('Can not tear down snapshot: %s' % device)
scrubbed = True
- # Now cleanup any safekeep directories still hanging around
+ # Now cleanup any safekeep directories and script files still hanging around
debug("Cleaning up remaining safekeep directories")
if os.path.isdir('/mnt'):
@@ -905,6 +920,33 @@
except OSError, e:
warn('Failed to remove: %s: %s' % (mountpoint, e))
+ debug("Cleaning up remaining safekeep script files")
+ script_dir = tempfile.gettempdir()
+ if os.path.isdir(script_dir):
+ for ent in os.listdir(script_dir):
+ if fnmatch.fnmatch(ent, 'safekeep-[0-9][0-9]*-') and not fnmatch.fnmatch(ent, 'safekeep-%d-' % current_pid):
+ script_file = '%s/%s' % (script_dir, ent)
+ if os.path.isdir(script_file):
+ temp_dir = script_file
+ for ent in os.listdir(temp_dir):
+ script_file = '%s/%s' % (temp_dir, ent)
+ info("Removing script file %s" % script_file)
+ try:
+ os.remove('%s' % script_file)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (script_file, e))
+ info("Removing script directory %s" % temp_dir)
+ try:
+ os.rmdir(temp_dir)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (temp_dir, e))
+ else:
+ info("Removing script file %s" % script_file)
+ try:
+ os.remove('%s' % script_file)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (script_file, e))
+
if not scrubbed:
info('No cleanup required')
@@ -921,6 +963,8 @@
bdir = '/'
cfg = do_client_config_parse('<backup/>', 'def')
ex = None
+ script_file = None
+ script_dir = None
try:
while True:
try:
@@ -930,6 +974,29 @@
send('OK %s, %s' % (PROTOCOL, VERSION))
elif line.startswith('CONFIG'):
cfg = do_client_config(line)
+ if ':' in cfg['script']:
+ (script_loc, script) = cfg['script'].split(':', 1)
+ else:
+ (script_loc, script) = ('client', cfg['script'])
+ if script_loc == 'server':
+ if not script_dir:
+ script_dir = tempfile.mkdtemp(prefix="safekeep-%d-" % current_pid)
+ script = os.path.basename(script)
+ (fd, cfg['script']) = tempfile.mkstemp(prefix="%s-" % script, dir=script_dir)
+ script_file = os.fdopen(fd, 'w')
+ send('OK %s' % cfg['script'])
+ elif script_loc == 'client':
+ cfg['script'] = script
+ ret = client_side_script('STARTUP', cfg, bdir)
+ if ret:
+ send('ERROR Client-side setup script failed: %s' % ret)
+ else:
+ send('OK')
+ else:
+ warn('Unknown script location %s for script %s' % (script_loc, script))
+ send('OK')
+ elif line.startswith('SCRIPT'):
+ do_client_remote_script(script_file, cfg, line)
ret = client_side_script('STARTUP', cfg, bdir)
if ret:
send('ERROR Client-side setup script failed: %s' % ret)
@@ -961,6 +1028,14 @@
finally:
if should_cleanup:
do_client_cleanup(cfg, bdir)
+ if script_file:
+ if not script_file.closed: script_file.close()
+ os.remove(cfg['script'])
+ if script_dir:
+ try:
+ os.rmdir(script_dir)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (script_dir, e))
if ex:
send('TRACEBACK ' + str(ex) + '>>>' + stacktrace().replace('\n', '###'))
@@ -1156,7 +1231,17 @@
cin.write('CONFIG: %d: %s\n' % (len(cfg['text'].splitlines()), id))
cin.write(cfg['text'] + '\n')
cin.flush()
- do_server_getanswer(cout)
+ remote_script = do_server_getanswer(cout)
+ if cfg['script'].startswith('server:') and remote_script:
+ local_script = cfg['script'].split(':', 1)[1]
+ debug("Transferring script: %s -> %s" % (local_script, remote_script))
+ fscript = open(local_script)
+ lines = fscript.readlines()
+ fscript.close()
+ cin.write('SCRIPT: %s: %d\n' % (local_script, len(lines)))
+ cin.writelines(lines)
+ cin.flush()
+ do_server_getanswer(cout)
if cleanup:
cleaned_up = False
cin.write('SCRUB\n')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-30 08:50:31
|
Revision: 783
http://safekeep.svn.sourceforge.net/safekeep/?rev=783&view=rev
Author: fcrawford
Date: 2011-12-30 08:50:25 +0000 (Fri, 30 Dec 2011)
Log Message:
-----------
Only add trailing slash on directory if it is not already present
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-30 08:43:51 UTC (rev 782)
+++ safekeep/trunk/safekeep 2011-12-30 08:50:25 UTC (rev 783)
@@ -724,7 +724,7 @@
def mount_excluded(cfg, mountpoint):
debug("mount_excluded: %s" % mountpoint)
- mountpoint = mountpoint + '/'
+ if not mountpoint.endswith('/'): mountpoint = mountpoint + '/'
for clude in cfg['cludes']:
if clude['type'] == 'exclude' and clude['path']:
if mountpoint.startswith(clude['path']):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-30 08:43:57
|
Revision: 782
http://safekeep.svn.sourceforge.net/safekeep/?rev=782&view=rev
Author: fcrawford
Date: 2011-12-30 08:43:51 +0000 (Fri, 30 Dec 2011)
Log Message:
-----------
Added LVM snapshot LV tagging support as requested by Andres Toomsalu
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/safekeep
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2011-12-25 13:40:47 UTC (rev 781)
+++ safekeep/trunk/doc/safekeep.backup.txt 2011-12-30 08:43:51 UTC (rev 782)
@@ -242,6 +242,12 @@
of the original device's size.
Mandatory for a `<snapshot>` element.
+/backup/setup/snapshot/@tag::
+ A tag to be added to the snapshot, with the `--addtag` argument
+ to `lvcreate`. An `@` is automatically added to the generated
+ tag.
+ Optional for a `<snapshot>` element.
+
/backup/setup/script/@path::
Execute the script specified path on the client at certain steps
of the backup process.
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-25 13:40:47 UTC (rev 781)
+++ safekeep/trunk/safekeep 2011-12-30 08:43:51 UTC (rev 782)
@@ -328,7 +328,8 @@
size = snap_el.getAttribute('size')
if not size:
raise ConfigException('Please specify the size for the snapshot')
- return { 'device' : device, 'size' : size }
+ tag = snap_el.getAttribute('tag')
+ return { 'device' : device, 'size' : size, 'tag' : tag }
def parse_clude(clude_el):
path = clude_el.getAttribute('path')
@@ -681,8 +682,14 @@
if not snapmnt:
warn('Cannot find the mountpoint for: %s' % device)
continue
- args = ['lvcreate', '--snapshot', '--size', snap['size'],
- '--name', os.path.basename(snapdev), lvmdev]
+ tag = snap['tag']
+ if tag:
+ args = ['lvcreate', '--addtag', '@' + tag,
+ '--snapshot', '--size', snap['size'],
+ '--name', os.path.basename(snapdev), lvmdev]
+ else:
+ args = ['lvcreate', '--snapshot', '--size', snap['size'],
+ '--name', os.path.basename(snapdev), lvmdev]
ec = spawn(args)
if ec:
warn('Can not snapshot the device: %s' % device)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-25 13:40:54
|
Revision: 781
http://safekeep.svn.sourceforge.net/safekeep/?rev=781&view=rev
Author: fcrawford
Date: 2011-12-25 13:40:47 +0000 (Sun, 25 Dec 2011)
Log Message:
-----------
Alexander List <al...@li...>
Typo in safekeep.cron creates file "1" instead of redirecting stderr to stdout.
Modified Paths:
--------------
safekeep/trunk/safekeep.cron
Modified: safekeep/trunk/safekeep.cron
===================================================================
--- safekeep/trunk/safekeep.cron 2011-12-07 15:18:46 UTC (rev 780)
+++ safekeep/trunk/safekeep.cron 2011-12-25 13:40:47 UTC (rev 781)
@@ -2,6 +2,6 @@
PATH=/sbin:/bin:/usr/sbin:/usr/bin
-if ls /etc/safekeep/backup.d/*.backup 2>1 1>/dev/null; then
+if ls /etc/safekeep/backup.d/*.backup 2>&1 1>/dev/null; then
safekeep -v --server
fi
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2011-12-07 15:18:57
|
Revision: 780
http://safekeep.svn.sourceforge.net/safekeep/?rev=780&view=rev
Author: dimi
Date: 2011-12-07 15:18:46 +0000 (Wed, 07 Dec 2011)
Log Message:
-----------
Appease ASCIIDOC, provide the required metadata.
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/doc/safekeep.conf.txt
safekeep/trunk/doc/safekeep.txt
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2011-12-07 03:38:00 UTC (rev 779)
+++ safekeep/trunk/doc/safekeep.backup.txt 2011-12-07 15:18:46 UTC (rev 780)
@@ -1,3 +1,7 @@
+:man source: safekeep
+:man version: {revnumber}
+:man manual: SafeKeep Client Configuration
+
safekeep.backup(5)
================
@@ -377,3 +381,6 @@
--------
safekeep(1), safekeep.conf(5), rdiff-backup(1), lvcreate(8)
+AUTHOR
+------
+This man page was written by Dimi Paun <di...@la...>.
Modified: safekeep/trunk/doc/safekeep.conf.txt
===================================================================
--- safekeep/trunk/doc/safekeep.conf.txt 2011-12-07 03:38:00 UTC (rev 779)
+++ safekeep/trunk/doc/safekeep.conf.txt 2011-12-07 15:18:46 UTC (rev 780)
@@ -1,3 +1,7 @@
+:man source: safekeep
+:man version: {revnumber}
+:man manual: SafeKeep Server Configuration
+
safekeep.conf(5)
================
@@ -120,3 +124,7 @@
--------
safekeep(1), safekeep.backup(5), rdiff-backup(1), trickle(1), lvcreate(8)
+AUTHOR
+------
+This man page was written by Dimi Paun <di...@la...>.
+
Modified: safekeep/trunk/doc/safekeep.txt
===================================================================
--- safekeep/trunk/doc/safekeep.txt 2011-12-07 03:38:00 UTC (rev 779)
+++ safekeep/trunk/doc/safekeep.txt 2011-12-07 15:18:46 UTC (rev 780)
@@ -1,3 +1,7 @@
+:man source: safekeep
+:man version: {revnumber}
+:man manual: SafeKeep Manual
+
safekeep(1)
==========
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2011-12-07 03:38:06
|
Revision: 779
http://safekeep.svn.sourceforge.net/safekeep/?rev=779&view=rev
Author: dimi
Date: 2011-12-07 03:38:00 +0000 (Wed, 07 Dec 2011)
Log Message:
-----------
Simpler, less overwhelming documentation
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2011-12-06 12:04:48 UTC (rev 778)
+++ safekeep/trunk/doc/safekeep.backup.txt 2011-12-07 03:38:00 UTC (rev 779)
@@ -22,50 +22,28 @@
------------------------------------------------------------------------
<backup id="my_workstation">
- <!-- the client backup host, the user under which the servers will connect,
- the SSH keys used for control and data transfer -->
- <host
- name="myhost" user="root" nice="10"
- key-ctrl="/home/jdoe/.ssh/backup_id_dsa"
- key-data="/home/jdoe/.ssh/backup2_id_dsa"
- />
+ <!-- the client backup host, the user and keys for access -->
+ <host name="myhost" user="root" nice="10"
+ key-ctrl="/home/jdoe/.ssh/backup_id_dsa"
+ key-data="/home/jdoe/.ssh/backup2_id_dsa" />
- <!-- you can customize the bandwidth limit on a client basis,
- for example to 50KB/s download/upload -->
+ <!-- rate limit bandwidth (kB/s) on a client basis -->
<bandwidth overall="80" download="50" upload="50" />
- <!-- location where the backups will be stored on the server
- and for how long (D=days, W=weeks, M=months, or Y=years) -->
- <repo path="./data" retention="10D"/>
+ <!-- where to stored the data on the server, and for now long -->
+ <repo path="./data" retention="10D" />
<!-- settings for database dump and for volume snapshot -->
<setup>
- <!-- database type ("postgres" or "mysql"), and database name,
- DB user with backup rights, location of the dump file on the client
- host, and flag to remove or not the dump file after the backup is done.
- Databases can be dumped individually using a dump clause for each database. -->
- <dump
- type="postgres"
- db="my_db"
- dbuser="foobar"
- options="--schema=public"
- file="/var/backup/dumps/mydata"
- cleanup="true"
- />
+ <!-- databases can be dumbed before the backup -->
+ <dump type="postgres" db="my_db" dbuser="foobar" options="--schema=public"
+ file="/var/backup/dumps/mydata" cleanup="true" />
- <!-- what volume is to be snapshot (device location) and the size
- of the snapshot (free space must exist in the volume group) -->
- <snapshot
- device="/path/to/volume"
- size="500M"
- />
+ <!-- what volume is to be snapshot, and the size of the snapshot -->
+ <snapshot device="/path/to/volume" size="500M" />
- <!-- location of a script to be executed on the client at different
- stages of the run. It is called with three arguments:
- the step of the backup, backup id and the backup root directory -->
- <script
- path="/path/to/script"
- />
+ <!-- location of a script to be executed at different stages of the run -->
+ <script path="/path/to/script" />
</setup>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-06 12:04:54
|
Revision: 778
http://safekeep.svn.sourceforge.net/safekeep/?rev=778&view=rev
Author: fcrawford
Date: 2011-12-06 12:04:48 +0000 (Tue, 06 Dec 2011)
Log Message:
-----------
Patch that will correct issues with performing
"mount --rbind / /mnt/safekeep...".
Instead of recursively binding the entire tree it only binds filesystems
that are mounted on "real" devices (i.e. have a device in the filesystem)
and that are not in the exclude list for the backup.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-03 02:36:01 UTC (rev 777)
+++ safekeep/trunk/safekeep 2011-12-06 12:04:48 UTC (rev 778)
@@ -715,6 +715,33 @@
if ret:
warn('Can not tear down snapshot: %s' % device)
+def mount_excluded(cfg, mountpoint):
+ debug("mount_excluded: %s" % mountpoint)
+ mountpoint = mountpoint + '/'
+ for clude in cfg['cludes']:
+ if clude['type'] == 'exclude' and clude['path']:
+ if mountpoint.startswith(clude['path']):
+ debug("mount_excluded: %s: matched %s" % (mountpoint, clude['path']))
+ return True
+ debug("mount_excluded: %s: no matches" % mountpoint)
+ return False
+
+def do_rbind(cfg, startpath, bdir):
+ for (device, mountpoint, mounttype, mountoptions) in mount_information(False):
+ debug("Testing %s on %s" % (mountpoint, device))
+ if mountpoint.startswith(startpath) and device.startswith('/'):
+ if not mount_excluded(cfg, mountpoint):
+ ret = spawn(['mount', '--bind', mountpoint, reroot(bdir, mountpoint)])
+ if ret:
+ debug("mount --bind %s: failed: unwinding" % mountpoint)
+ ret = spawn(['umount', '-l', reroot(bdir, startpath)])
+ if ret:
+ warn('Failed to unmount: %s' % reroot(bdir, startpath))
+ return 1
+ spawn(['mount', '--make-unbindable', reroot(bdir, mountpoint)])
+
+ return 0
+
######################################################################
# Client implementation
######################################################################
@@ -753,7 +780,7 @@
if ret:
warn('modprobe dm-snapshot failed, continuing')
bdir = tempfile.mkdtemp("-rbind", "safekeep-%d-" % current_pid, "/mnt")
- ret = spawn(['mount', '--rbind', '/', bdir])
+ ret = do_rbind(cfg, '/', bdir)
if ret:
warn('mount --rbind failed, snapshotting will be disabled')
try:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-03 02:36:08
|
Revision: 777
http://safekeep.svn.sourceforge.net/safekeep/?rev=777&view=rev
Author: fcrawford
Date: 2011-12-03 02:36:01 +0000 (Sat, 03 Dec 2011)
Log Message:
-----------
Futher updates to cleanup code for failed backups
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-12-02 12:51:50 UTC (rev 776)
+++ safekeep/trunk/safekeep 2011-12-03 02:36:01 UTC (rev 777)
@@ -649,13 +649,17 @@
lvmdev = device;
else:
lvmdev = '/dev/mapper/%s-%s' % (group, volume.replace('-', '--'))
- for i in range(1,10):
- ret = spawn(['sync'])
- ret = spawn(['dmsetup', 'remove', lvmdev])
- ret = spawn(['dmsetup', 'remove', lvmdev + '-cow'])
- ret = spawn(['lvremove', '--force', device])
- if not ret:
- break
+ if os.path.exists(lvmdev):
+ for i in range(1,10):
+ ret = spawn(['sync'])
+ ret = spawn(['dmsetup', 'remove', lvmdev])
+ ret = spawn(['dmsetup', 'remove', lvmdev + '-cow'])
+ ret = spawn(['lvremove', '--force', device])
+ if not ret:
+ break
+ else:
+ warn('lvremove called for non-existent device: %s' % lvmdev)
+ ret = 0 # Equivalent to lvremove succeeding
return ret
def gather_snap_information(device, bdir):
@@ -798,8 +802,7 @@
# Go through and see if any come from existing safekeep processes
pattern = re.compile(r"_snap_safekeep-(\d+)-")
- lvm_snap_list = lvm_snap_information()
- for (volume, group) in lvm_snap_list:
+ for (volume, group) in lvm_snap_information():
matches = pattern.search(volume)
if matches is not None:
pid = matches.group(1)
@@ -847,7 +850,7 @@
# Now cleanup any snapshots still hanging around
debug("Cleaning up remaining snapshots")
- for (volume, group) in lvm_snap_list:
+ for (volume, group) in lvm_snap_information():
device = os.path.join('/dev', group, volume)
info("Removing snapshot %s" % device)
ret = do_lvremove(device)
@@ -1078,7 +1081,7 @@
info('Server backup starting for client %s' % id)
output_done = True
- cleaned_up = False
+ cleaned_up = True
try:
if cfg['host']:
if not os.path.isfile(cfg['key_ctrl']):
@@ -1121,6 +1124,7 @@
cin.flush()
do_server_getanswer(cout)
if cleanup:
+ cleaned_up = False
cin.write('SCRUB\n')
cin.flush()
do_server_getanswer(cout)
@@ -1145,6 +1149,7 @@
else:
backup_marker = None
+ cleaned_up = False
do_server_rdiff(cfg, bdir, nice, ionice, force)
cleaned_up = True
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-12-02 12:51:58
|
Revision: 776
http://safekeep.svn.sourceforge.net/safekeep/?rev=776&view=rev
Author: fcrawford
Date: 2011-12-02 12:51:50 +0000 (Fri, 02 Dec 2011)
Log Message:
-----------
Joe Steele <jo...@ma...>
Maintain compatibility with Python 2.2
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-11-28 12:38:37 UTC (rev 775)
+++ safekeep/trunk/safekeep 2011-12-02 12:51:50 UTC (rev 776)
@@ -19,7 +19,7 @@
import getopt, os, os.path, re, sys, fnmatch, stat, types
import commands, tempfile, time, traceback
import getpass, pwd, xml.dom.minidom
-import socket, smtplib
+import socket, smtplib, cStringIO
try:
import subprocess
@@ -128,7 +128,9 @@
return errs
def stacktrace():
- return "\n" + traceback.format_exc()
+ exc_file = cStringIO.StringIO()
+ traceback.print_exc(None, exc_file)
+ return "\n" + exc_file.getvalue()
def debug(msg):
log(msg, 'DBG')
@@ -171,7 +173,8 @@
if _stderr:
(child_in, child_out) = os.popen4(args)
else:
- (child_in, child_out) = os.popen3(args)
+ (child_in, child_out, child_err) = os.popen3(args)
+ child_err.close()
if not stdin:
child_in.close()
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-11-28 12:38:43
|
Revision: 775
http://safekeep.svn.sourceforge.net/safekeep/?rev=775&view=rev
Author: fcrawford
Date: 2011-11-28 12:38:37 +0000 (Mon, 28 Nov 2011)
Log Message:
-----------
- includes the PID of the creating safekeep in the snapshot names,
- handles the client side cleanup, if snapshots are left around,
- cleans up the backup files if the client fails,
- intentionally block run a "safekeep --client --cleanup" when other safekeep is running.
- clean up of messages and some control flow,
- suppress tracebacks, only printing them on debug requests,
- ensure that the client is closed before starting some possibly long running server processes.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2011-11-28 12:38:02 UTC (rev 774)
+++ safekeep/trunk/safekeep 2011-11-28 12:38:37 UTC (rev 775)
@@ -64,6 +64,7 @@
backup_user = None
home_dir = None
base_dir = None
+current_pid = os.getpid()
default_bandwidth = {}
cmd = "<Missing>"
@@ -140,7 +141,7 @@
def error(msg, ex=None):
extra = ""
- if ex:
+ if ex and verbosity_level > 2:
extra = stacktrace()
log(msg + extra, 'ERR')
@@ -595,7 +596,7 @@
continue
try:
os.remove(dump['file'])
- except Exception, e:
+ except OSError, e:
warn('Unable to remove dump file: %s for database %s because: %s' %
(dump['file'], dump['db'], e))
@@ -738,20 +739,20 @@
debug('Checking FS snapshots')
for snap in cfg['snaps']:
device = snap['device']
- if check_lvm_information(device):
- raise Exception("Previous snapshots found for %s: run 'safekeep --server --cleanup' to correct" % device)
+ if check_lvm_information(device) and not do_client_scrub():
+ raise Exception("Previous snapshots found for %s and automatic correction failed: run 'safekeep --server --cleanup' to correct" % device)
ret = spawn(['modprobe', 'dm-snapshot'])
if ret:
warn('modprobe dm-snapshot failed, continuing')
- bdir = tempfile.mkdtemp("-rbind", "safekeep-", "/mnt")
+ bdir = tempfile.mkdtemp("-rbind", "safekeep-%d-" % current_pid, "/mnt")
ret = spawn(['mount', '--rbind', '/', bdir])
if ret:
warn('mount --rbind failed, snapshotting will be disabled')
try:
os.rmdir(bdir)
- except Exception, e:
- warn('Failed to remove: %s' % bdir)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (bdir, e))
bdir = '/'
else:
do_client_snap(cfg, bdir)
@@ -772,8 +773,8 @@
else:
try:
os.rmdir(bdir)
- except Exception, e:
- warn('Unable to remove: ' + bdir)
+ except OSError, e:
+ warn('Unable to remove: %s: %s' % (bdir, e))
do_client_dbdump_teardown(cfg)
@@ -792,6 +793,22 @@
else:
scrubbed = False
+ # Go through and see if any come from existing safekeep processes
+ pattern = re.compile(r"_snap_safekeep-(\d+)-")
+ lvm_snap_list = lvm_snap_information()
+ for (volume, group) in lvm_snap_list:
+ matches = pattern.search(volume)
+ if matches is not None:
+ pid = matches.group(1)
+ # Look up /proc/<pid>/cmdline to see what process is running
+ proc_file = "/proc/" + pid + "/cmdline"
+ if pid != current_pid and os.path.exists(proc_file):
+ fin = open(proc_file, "r")
+ (cmd, arg0, args) = fin.read().split('\0', 2)
+ fin.close()
+ if os.path.basename(arg0) == "safekeep":
+ raise Exception('another safekeep process running: pid %s' % pid)
+
if os.environ['PATH'][-1] == ':':
os.environ['PATH'] += '/sbin:/usr/sbin:/usr/local/sbin:'
else:
@@ -811,8 +828,8 @@
else:
try:
os.rmdir(mountpoint)
- except Exception, e:
- warn('Failed to remove: %s' % mountpoint)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (mountpoint, e))
else:
ret = spawn(['umount', mountpoint])
if ret:
@@ -827,7 +844,7 @@
# Now cleanup any snapshots still hanging around
debug("Cleaning up remaining snapshots")
- for (volume, group) in lvm_snap_information():
+ for (volume, group) in lvm_snap_list:
device = os.path.join('/dev', group, volume)
info("Removing snapshot %s" % device)
ret = do_lvremove(device)
@@ -845,12 +862,19 @@
info("Removing rbind directory %s" % mountpoint)
try:
os.rmdir(mountpoint)
- except Exception, e:
- warn('Failed to remove: %s' % mountpoint)
+ except OSError, e:
+ warn('Failed to remove: %s: %s' % (mountpoint, e))
if not scrubbed:
info('No cleanup required')
+ # This has to be rerun to see if it has been successful
+ if lvm_snap_information():
+ return False
+ return True
+
+ return False
+
def do_client():
debug("Do client main loop")
should_cleanup = True
@@ -914,7 +938,7 @@
raise ClientException(line[5:].strip())
elif line.startswith('TRACEBACK'):
i = line.find('>>>')
- raise ClientException(line[10:i].strip(), line[i+3:].replace('###', '\n'))
+ raise ClientException(line[10:i].strip(), line[i+3:].replace('###', '\n').rstrip())
elif not line:
raise Exception('client died unexpectedly')
else:
@@ -1064,7 +1088,7 @@
try:
os.makedirs(datadir)
except EnvironmentError, ex:
- raise Exception('Can not create data store dir: %s' % datadir)
+ raise Exception('Can not create data store dir: %s: %s' % (datadir, ex))
rdiff_logdir = os.path.join(datadir, 'rdiff-backup-data')
if cfg['retention'] and os.path.isdir(rdiff_logdir) and not cleanup:
@@ -1098,8 +1122,6 @@
cin.flush()
do_server_getanswer(cout)
bdir = '/' # Fake directory for the rest of the cleanup
- do_server_rdiff_cleanup(cfg)
- cleaned_up = True
errs = 0
else:
cin.write('SETUP\n')
@@ -1121,6 +1143,7 @@
backup_marker = None
do_server_rdiff(cfg, bdir, nice, ionice, force)
+ cleaned_up = True
errs = 0
if os.path.isdir(rdiff_logdir):
@@ -1142,16 +1165,23 @@
info('Server backup for client %s: OK (%d WARNINGS)' % (id, errs))
except Exception, ex:
- if cleanup and not cleaned_up:
+ if cleanup:
info('Client-side cleanup for client %s: FAILED' % id)
- do_server_rdiff_cleanup(cfg)
else:
if isinstance(ex, ClientException):
error('Client %s: FAILED due to: %s' % (id, ex or ''))
- if ex.traceback: error(ex.traceback)
+ if ex.traceback and verbosity_level > 2: error(ex.traceback)
else:
error('Server backup for client %s: FAILED' % id, ex)
+ # Shutdown client
+ cout.close()
+ cin.close()
+
+ if not cleaned_up:
+ do_server_rdiff_cleanup(cfg)
+ cleaned_up = True
+
if output_done:
info('------------------------------------------------------------------')
debug('Server backup done')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fcr...@us...> - 2011-11-28 12:38:08
|
Revision: 774
http://safekeep.svn.sourceforge.net/safekeep/?rev=774&view=rev
Author: fcrawford
Date: 2011-11-28 12:38:02 +0000 (Mon, 28 Nov 2011)
Log Message:
-----------
Changes to work with recent release
Modified Paths:
--------------
safekeep/trunk/Makefile
Modified: safekeep/trunk/Makefile
===================================================================
--- safekeep/trunk/Makefile 2011-11-27 02:50:38 UTC (rev 773)
+++ safekeep/trunk/Makefile 2011-11-28 12:38:02 UTC (rev 774)
@@ -13,12 +13,13 @@
svnroot := $(shell LANG=C svn info 2>/dev/null | grep Root | cut -c 18-)
deb_box := 192.168.3.202
rpm_box := 192.168.3.242
-sf_login := dimi,$(name)@frs.sourceforge.net
+SF_USER := $(shell whoami)
+sf_login := $(SF_USER),$(name)@frs.sourceforge.net
sf_dir := /home/frs/project/s/sa/$(name)/$(name)
releasedir := releases
repo_srv := root@ulysses
repo_dir := /var/www/repos/lattica
-webroot := ../website/WebContent/
+webroot := ../../website/trunk/WebContent/
MAN_TXT := doc/safekeep.txt doc/safekeep.conf.txt doc/safekeep.backup.txt
DOC_MAN := doc/safekeep.1 doc/safekeep.conf.5 doc/safekeep.backup.5
DOC_HTML := $(patsubst %.txt,%.html,$(MAN_TXT))
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
