Menu
▾
▴
safekeep-commit — Source code changes (CVS commits) for SafeKeep
You can subscribe to this list here.
| 2007 |
Jan
(76) |
Feb
(76) |
Mar
(54) |
Apr
(14) |
May
(23) |
Jun
(21) |
Jul
|
Aug
|
Sep
(9) |
Oct
(14) |
Nov
(12) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2008 |
Jan
|
Feb
(18) |
Mar
(3) |
Apr
|
May
|
Jun
(2) |
Jul
(3) |
Aug
|
Sep
|
Oct
(17) |
Nov
(13) |
Dec
|
| 2009 |
Jan
(1) |
Feb
(1) |
Mar
(15) |
Apr
(2) |
May
(18) |
Jun
(1) |
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2010 |
Jan
(1) |
Feb
(5) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(61) |
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
(18) |
Apr
|
May
|
Jun
(5) |
Jul
|
Aug
(1) |
Sep
(2) |
Oct
(4) |
Nov
(10) |
Dec
(9) |
| 2012 |
Jan
(10) |
Feb
(23) |
Mar
|
Apr
(5) |
May
(2) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
(4) |
| 2013 |
Jan
(17) |
Feb
(1) |
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <di...@us...> - 2010-11-19 17:17:43
|
Revision: 698
http://safekeep.svn.sourceforge.net/safekeep/?rev=698&view=rev
Author: dimi
Date: 2010-11-19 17:17:37 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Cleanup error handling
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 17:01:47 UTC (rev 697)
+++ safekeep/trunk/safekeep 2010-11-19 17:17:37 UTC (rev 698)
@@ -778,6 +778,7 @@
should_cleanup = True
bdir = '/'
cfg = do_client_config_parse('<backup/>', 'def')
+ ex = None
try:
while True:
try:
@@ -813,12 +814,14 @@
send('ERROR Unknown command: %s' % line)
break
except Exception, e:
- traceback.print_exc(file=sys.stdout)
- send('ERROR %s' % e)
+ ex = e
+ break
finally:
if should_cleanup:
do_client_cleanup(cfg, bdir)
+ if ex:
+ send('ERROR %s' % (ex or ''))
######################################################################
# Server implementation
@@ -1482,8 +1485,7 @@
else:
assert False, 'Unknown mode: %s' % (mode)
except Exception, ex:
- traceback.print_exc(file=sys.stdout)
- error('ERROR: %s' % ex)
+ error('ERROR: %s' % (ex or ''))
if email and not noemail:
send_notification(email, smtp)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 17:01:53
|
Revision: 697
http://safekeep.svn.sourceforge.net/safekeep/?rev=697&view=rev
Author: dimi
Date: 2010-11-19 17:01:47 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
We have to deal with strings here
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:59:55 UTC (rev 696)
+++ safekeep/trunk/safekeep 2010-11-19 17:01:47 UTC (rev 697)
@@ -105,7 +105,7 @@
log(msg, 'ERR')
def do_spawn(args, stdin=None, stdout=False):
- debug('Run [' + args + ']')
+ debug('Run [' + ' '.joint(args) + ']')
_shell = isinstance(args, types.StringTypes)
if stdin:
_stdin = subprocess.PIPE
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 17:00:03
|
Revision: 696
http://safekeep.svn.sourceforge.net/safekeep/?rev=696&view=rev
Author: dimi
Date: 2010-11-19 16:59:55 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Better handle errors in case things go really bad
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:50:18 UTC (rev 695)
+++ safekeep/trunk/safekeep 2010-11-19 16:59:55 UTC (rev 696)
@@ -642,6 +642,9 @@
if not line: raise ConfigException('Unexpected end of file')
cfgStr += line
+ do_client_config_parse(cfgStr, dflt_id)
+
+def do_client_config_parse(cfgStr, dflt_id=None):
dom = xml.dom.minidom.parseString(cfgStr)
try:
return parse_config(dom.documentElement, dflt_id)
@@ -774,6 +777,7 @@
debug("Do client main loop")
should_cleanup = True
bdir = '/'
+ cfg = do_client_config_parse('<backup/>', 'def')
try:
while True:
try:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:50:24
|
Revision: 695
http://safekeep.svn.sourceforge.net/safekeep/?rev=695&view=rev
Author: dimi
Date: 2010-11-19 16:50:18 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
More cleanup of the subprocess usage
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:44:13 UTC (rev 694)
+++ safekeep/trunk/safekeep 2010-11-19 16:50:18 UTC (rev 695)
@@ -22,8 +22,6 @@
import getpass, pwd, xml.dom.minidom
import socket, smtplib
-from subprocess import PIPE, STDOUT
-
######################################################################
# Global settings
######################################################################
@@ -108,12 +106,20 @@
def do_spawn(args, stdin=None, stdout=False):
debug('Run [' + args + ']')
- shell = isinstance(args, types.StringTypes)
- proc = subprocess.Popen(args, bufsize=1, shell=shell, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ _shell = isinstance(args, types.StringTypes)
+ if stdin:
+ _stdin = subprocess.PIPE
+ else:
+ _stdin = None
+ if stdout:
+ _stderr = None
+ else:
+ _stderr = subprocess.STDOUT
+ proc = subprocess.Popen(args, bufsize=1, shell=_shell, stdin=_stdin, stdout=subprocess.PIPE, stderr=_stderr, close_fds=True)
if stdin:
proc.stdin.write(stdin)
- proc.stdin.close()
+ proc.stdin.close()
lines=[]
for line in proc.stdout:
if stdout:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:44:19
|
Revision: 694
http://safekeep.svn.sourceforge.net/safekeep/?rev=694&view=rev
Author: dimi
Date: 2010-11-19 16:44:13 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Cleanup
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:43:36 UTC (rev 693)
+++ safekeep/trunk/safekeep 2010-11-19 16:44:13 UTC (rev 694)
@@ -124,7 +124,7 @@
return (proc.wait(), ''.join(lines))
def _spawn(args, stdin=None, stdout=False):
- if isinstance(args, str) or isinstance(args, unicode):
+ if isinstance(args, types.StringTypes):
cmd = args.split(None)[0]
else:
cmd = args[0]
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:43:42
|
Revision: 693
http://safekeep.svn.sourceforge.net/safekeep/?rev=693&view=rev
Author: dimi
Date: 2010-11-19 16:43:36 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Auto "shell" detection: if the command is a string,
it will be executed via the shell, otherwise it will
be executed directly.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:36:52 UTC (rev 692)
+++ safekeep/trunk/safekeep 2010-11-19 16:43:36 UTC (rev 693)
@@ -16,7 +16,7 @@
# along with Safekeep. If not, see <http://www.gnu.org/licenses/>.
from __future__ import generators
-import getopt, os, os.path, re, sys, fnmatch, stat
+import getopt, os, os.path, re, sys, fnmatch, stat, types
import subprocess
import commands, tempfile, time, traceback
import getpass, pwd, xml.dom.minidom
@@ -106,8 +106,9 @@
def error(msg):
log(msg, 'ERR')
-def do_spawn(args, shell=False, stdin=None, stdout=False):
+def do_spawn(args, stdin=None, stdout=False):
debug('Run [' + args + ']')
+ shell = isinstance(args, types.StringTypes)
proc = subprocess.Popen(args, bufsize=1, shell=shell, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
if stdin:
@@ -122,14 +123,14 @@
proc.stdout.close()
return (proc.wait(), ''.join(lines))
-def _spawn(args, shell=False, stdin=None, stdout=False):
+def _spawn(args, stdin=None, stdout=False):
if isinstance(args, str) or isinstance(args, unicode):
cmd = args.split(None)[0]
else:
cmd = args[0]
try:
- rc, out = do_spawn(args, shell, stdin, stdout)
+ rc, out = do_spawn(args, stdin, stdout)
except OSError, ex:
ret = "OSError: %s" % (ex)
error('%s failed: %s' % (cmd, ret));
@@ -149,15 +150,15 @@
# this just spawns an external program (optionally through a shell)
# and returns True it it fails, and False if it successed
-def spawn(args, shell=False):
- rc, out = _spawn(args, shell)
+def spawn(args):
+ rc, out = _spawn(args)
return rc
# this spawans an external program (optionally through a shell),
# feeds it any input via stdin, captures the output and returns it.
# if it fails it returns None, otherwise it returns the output
-def call(args, shell=False, stdin=None):
- rc, out = _spawn(args, shell, stdin, stdout=True)
+def call(args, stdin=None):
+ rc, out = _spawn(args, stdin, stdout=True)
if not rc:
return None
return out
@@ -505,7 +506,7 @@
if passwdfile:
os.environ['PGPASSFILE'] = passwdfile
try:
- ec = spawn(cmd, shell=True)
+ ec = spawn(cmd)
finally:
if passwdfile:
del os.environ['PGPASSFILE']
@@ -1149,7 +1150,7 @@
if deploy:
cmd = '%s %s@%s "umask 077; test -d .ssh || mkdir .ssh; cat >> .ssh/authorized_keys"' % (basessh, cfg['user'], cfg['host'])
keys = '%s\n' % '\n'.join([key[4] for key in new_keys])
- out = call(cmd, shell=True, stdin=keys)
+ out = call(cmd, stdin=keys)
if out is None:
error('Failed to deliver the keys to the client')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:36:58
|
Revision: 692
http://safekeep.svn.sourceforge.net/safekeep/?rev=692&view=rev
Author: dimi
Date: 2010-11-19 16:36:52 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Typo
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:35:45 UTC (rev 691)
+++ safekeep/trunk/safekeep 2010-11-19 16:36:52 UTC (rev 692)
@@ -955,7 +955,7 @@
cmd = []
if cfg['host']:
- cmd.extend(['ssh', verbosity_ssh, '-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host'])
+ cmd.extend(['ssh', verbosity_ssh, '-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host']])
cmd.extend(['safekeep', '--client'])
(cin,cout) = os.popen2(cmd)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:35:51
|
Revision: 691
http://safekeep.svn.sourceforge.net/safekeep/?rev=691&view=rev
Author: dimi
Date: 2010-11-19 16:35:45 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Get rid of the direct subprocess call.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:22:38 UTC (rev 690)
+++ safekeep/trunk/safekeep 2010-11-19 16:35:45 UTC (rev 691)
@@ -957,10 +957,8 @@
if cfg['host']:
cmd.extend(['ssh', verbosity_ssh, '-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host'])
cmd.extend(['safekeep', '--client'])
-
- proc = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- cin = proc.stdin
- cout = proc.stdout
+
+ (cin,cout) = os.popen2(cmd)
cin.write('ALOHA: %s, %s\n' % (PROTOCOL, VERSION))
cin.flush()
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:22:44
|
Revision: 690
http://safekeep.svn.sourceforge.net/safekeep/?rev=690&view=rev
Author: dimi
Date: 2010-11-19 16:22:38 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Get rid of a bunch of subprocess calls
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 16:13:21 UTC (rev 689)
+++ safekeep/trunk/safekeep 2010-11-19 16:22:38 UTC (rev 690)
@@ -184,11 +184,7 @@
server.quit()
else:
cmd = ['/usr/sbin/sendmail', '-t']
- pin = subprocess.Popen(cmd, stdin=PIPE).stdin
- try:
- pin.write(msg)
- finally:
- pin.close()
+ call(cmd, stdin=msg)
def is_temp_root(dir):
return dir != '/'
@@ -1134,10 +1130,8 @@
if status or deploy:
cmd = '%s %s@%s "if test -f .ssh/authorized_keys; then cat .ssh/authorized_keys; fi"' % (basessh, cfg['user'], cfg['host'])
- debug(cmd)
- out = subprocess.Popen(cmd, shell=True, stdout=PIPE).stdout
- authtext = out.read()
- if out.close():
+ authtext = call(cmd)
+ if authtext is None:
warn('%s: Failed to read the authorized_keys file.' % id)
auth_keys = parse_authorized_keys(authtext)
this_keys = parse_authorized_keys(output)
@@ -1156,10 +1150,9 @@
print '%s: Keys will be deployed on the client.' % id
if deploy:
cmd = '%s %s@%s "umask 077; test -d .ssh || mkdir .ssh; cat >> .ssh/authorized_keys"' % (basessh, cfg['user'], cfg['host'])
- debug(cmd)
- pipe = subprocess.Popen(cmd, shell=True, stdin=PIPE).stdin
- pipe.write('%s\n' % '\n'.join([key[4] for key in new_keys]))
- if pipe.close():
+ keys = '%s\n' % '\n'.join([key[4] for key in new_keys])
+ out = call(cmd, shell=True, stdin=keys)
+ if out is None:
error('Failed to deliver the keys to the client')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 16:13:32
|
Revision: 689
http://safekeep.svn.sourceforge.net/safekeep/?rev=689&view=rev
Author: dimi
Date: 2010-11-19 16:13:21 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Get rid of args_to_list()
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 15:07:44 UTC (rev 688)
+++ safekeep/trunk/safekeep 2010-11-19 16:13:21 UTC (rev 689)
@@ -106,24 +106,10 @@
def error(msg):
log(msg, 'ERR')
-def args_to_list(args):
- if isinstance(args, str) or isinstance(args, unicode):
- return args.split(None)
- else:
- return args
-
def do_spawn(args, shell=False, stdin=None, stdout=False):
- global cmd
- argslist = args_to_list(args)
- cmd = argslist[0]
- if shell:
- # If passed to a shell then give args exactly as specified.
- debug('Run [' + args + ']')
- proc = subprocess.Popen(args, bufsize=1, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- else:
- # Otherwise split into separate elements.
- debug('Run [' + ' '.join(argslist) + ']')
- proc = subprocess.Popen(argslist, bufsize=1, shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ debug('Run [' + args + ']')
+ proc = subprocess.Popen(args, bufsize=1, shell=shell, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+
if stdin:
proc.stdin.write(stdin)
proc.stdin.close()
@@ -137,7 +123,11 @@
return (proc.wait(), ''.join(lines))
def _spawn(args, shell=False, stdin=None, stdout=False):
- global cmd
+ if isinstance(args, str) or isinstance(args, unicode):
+ cmd = args.split(None)[0]
+ else:
+ cmd = args[0]
+
try:
rc, out = do_spawn(args, shell, stdin, stdout)
except OSError, ex:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 15:07:50
|
Revision: 688
http://safekeep.svn.sourceforge.net/safekeep/?rev=688&view=rev
Author: dimi
Date: 2010-11-19 15:07:44 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Get rid of cmd_run function.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 14:56:46 UTC (rev 687)
+++ safekeep/trunk/safekeep 2010-11-19 15:07:44 UTC (rev 688)
@@ -179,12 +179,6 @@
return False
return rc in (0,1)
-def cmd_run(args):
- argslist = args_to_list(args)
- debug('Run [' + ' '.join(argslist) + ']')
- p = subprocess.Popen(argslist, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- return (p.stdin, p.stdout)
-
def send_notification(email, smtp):
global logbuf
if not logbuf: return
@@ -545,10 +539,7 @@
(dump['file'], dump['db'], e))
def lvm_snap_information():
- (cin, cout) = cmd_run(['lvs', '--separator', ':', '--noheadings'])
- lines = cout.readlines()
- cout.close()
- cin.close()
+ lines = call(['lvs', '--separator', ':', '--noheadings']) or ''
lvms = []
for line in lines:
if line.count(':') > 3:
@@ -558,10 +549,7 @@
return lvms
def mount_information(reverse = False):
- (cin, cout) = cmd_run(['mount'])
- lines = cout.readlines()
- cout.close()
- cin.close()
+ lines = call(['mount']) or ''
mounts = []
pattern = re.compile(r"^(\S+) on (.+) type (\S+) \((\S+)\)")
if reverse:
@@ -979,11 +967,14 @@
if cfg['retention'] and os.path.isdir(rdiff_logdir) and not cleanup:
do_server_data_cleanup(cfg)
+ cmd = []
if cfg['host']:
- cmd = 'ssh %s -T -i %s -l %s %s safekeep --client' % (verbosity_ssh, cfg['key_ctrl'], cfg['user'], cfg['host'])
- else:
- cmd = 'safekeep --client'
- (cin, cout) = cmd_run(cmd)
+ cmd.extend(['ssh', verbosity_ssh, '-T', '-i', cfg['key_ctrl'], '-l', cfg['user'], cfg['host'])
+ cmd.extend(['safekeep', '--client'])
+
+ proc = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ cin = proc.stdin
+ cout = proc.stdout
cin.write('ALOHA: %s, %s\n' % (PROTOCOL, VERSION))
cin.flush()
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 14:56:52
|
Revision: 687
http://safekeep.svn.sourceforge.net/safekeep/?rev=687&view=rev
Author: dimi
Date: 2010-11-19 14:56:46 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Provide a more expressive function to call external commands
and deal with the standard input/output.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 14:41:04 UTC (rev 686)
+++ safekeep/trunk/safekeep 2010-11-19 14:56:46 UTC (rev 687)
@@ -112,7 +112,7 @@
else:
return args
-def do_spawn(args, shell=False, stdout=False):
+def do_spawn(args, shell=False, stdin=None, stdout=False):
global cmd
argslist = args_to_list(args)
cmd = argslist[0]
@@ -124,6 +124,8 @@
# Otherwise split into separate elements.
debug('Run [' + ' '.join(argslist) + ']')
proc = subprocess.Popen(argslist, bufsize=1, shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ if stdin:
+ proc.stdin.write(stdin)
proc.stdin.close()
lines=[]
for line in proc.stdout:
@@ -134,12 +136,14 @@
proc.stdout.close()
return (proc.wait(), ''.join(lines))
-def spawn(args, shell=False):
+def _spawn(args, shell=False, stdin=None, stdout=False):
global cmd
try:
- rc, out = do_spawn(args, shell=shell)
+ rc, out = do_spawn(args, shell, stdin, stdout)
except OSError, ex:
- error("OSError: %s: %s" % (cmd, ex))
+ ret = "OSError: %s" % (ex)
+ error('%s failed: %s' % (cmd, ret));
+ return ret
if not rc:
ret = None
@@ -151,8 +155,23 @@
ret = 'unknown exit status: %d' % rc
if ret:
error('%s failed: %s' % (cmd, ret));
- return ret
+ return (ret, out)
+# this just spawns an external program (optionally through a shell)
+# and returns True it it fails, and False if it successed
+def spawn(args, shell=False):
+ rc, out = _spawn(args, shell)
+ return rc
+
+# this spawans an external program (optionally through a shell),
+# feeds it any input via stdin, captures the output and returns it.
+# if it fails it returns None, otherwise it returns the output
+def call(args, shell=False, stdin=None):
+ rc, out = _spawn(args, shell, stdin, stdout=True)
+ if not rc:
+ return None
+ return out
+
def try_to_run(args):
try:
rc, out = do_spawn(args)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 14:41:10
|
Revision: 686
http://safekeep.svn.sourceforge.net/safekeep/?rev=686&view=rev
Author: dimi
Date: 2010-11-19 14:41:04 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Add ability to capture the output of a spawned proccess
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 08:57:27 UTC (rev 685)
+++ safekeep/trunk/safekeep 2010-11-19 14:41:04 UTC (rev 686)
@@ -112,7 +112,7 @@
else:
return args
-def do_spawn(args, shell=False):
+def do_spawn(args, shell=False, stdout=False):
global cmd
argslist = args_to_list(args)
cmd = argslist[0]
@@ -125,15 +125,19 @@
debug('Run [' + ' '.join(argslist) + ']')
proc = subprocess.Popen(argslist, bufsize=1, shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
proc.stdin.close()
+ lines=[]
for line in proc.stdout:
- info(line.rstrip())
+ if stdout:
+ lines.append(line)
+ else:
+ info(line.rstrip())
proc.stdout.close()
- return proc.wait()
+ return (proc.wait(), ''.join(lines))
def spawn(args, shell=False):
global cmd
try:
- rc = do_spawn(args, shell=shell)
+ rc, out = do_spawn(args, shell=shell)
except OSError, ex:
error("OSError: %s: %s" % (cmd, ex))
@@ -151,7 +155,7 @@
def try_to_run(args):
try:
- rc = do_spawn(args)
+ rc, out = do_spawn(args)
except OSError, ex:
return False
return rc in (0,1)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 08:57:33
|
Revision: 685
http://safekeep.svn.sourceforge.net/safekeep/?rev=685&view=rev
Author: dimi
Date: 2010-11-19 08:57:27 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Avoid subprocess where we can easily
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 08:34:35 UTC (rev 684)
+++ safekeep/trunk/safekeep 2010-11-19 08:57:27 UTC (rev 685)
@@ -149,9 +149,9 @@
error('%s failed: %s' % (cmd, ret));
return ret
-def try_to_run(args, shell=False):
+def try_to_run(args):
try:
- rc = do_spawn(args, shell=shell)
+ rc = do_spawn(args)
except OSError, ex:
return False
return rc in (0,1)
@@ -1104,7 +1104,7 @@
if backup_user is not work_user:
gencmd = 'su -s /bin/sh -c %s - %s' % (commands.mkarg(gencmd), backup_user)
debug(gencmd)
- if subprocess.Popen(gencmd, shell=True).wait():
+ if os.system(gencmd):
error('%s: Failed to generate key %s. Skipping client.' % (id, privatekeyfile))
break
if not os.path.isfile(publickeyfile):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 08:34:41
|
Revision: 684
http://safekeep.svn.sourceforge.net/safekeep/?rev=684&view=rev
Author: dimi
Date: 2010-11-19 08:34:35 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Use the Popen call directly
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 08:30:06 UTC (rev 683)
+++ safekeep/trunk/safekeep 2010-11-19 08:34:35 UTC (rev 684)
@@ -1104,7 +1104,7 @@
if backup_user is not work_user:
gencmd = 'su -s /bin/sh -c %s - %s' % (commands.mkarg(gencmd), backup_user)
debug(gencmd)
- if subprocess.call(gencmd, shell=True):
+ if subprocess.Popen(gencmd, shell=True).wait():
error('%s: Failed to generate key %s. Skipping client.' % (id, privatekeyfile))
break
if not os.path.isfile(publickeyfile):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 08:30:12
|
Revision: 683
http://safekeep.svn.sourceforge.net/safekeep/?rev=683&view=rev
Author: dimi
Date: 2010-11-19 08:30:06 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Simpler
Modified Paths:
--------------
safekeep/trunk/ANNOUNCE
Modified: safekeep/trunk/ANNOUNCE
===================================================================
--- safekeep/trunk/ANNOUNCE 2010-11-19 07:38:15 UTC (rev 682)
+++ safekeep/trunk/ANNOUNCE 2010-11-19 08:30:06 UTC (rev 683)
@@ -17,11 +17,9 @@
Thanks go to Frank Crawford and Bertrand Lecervoisier for providing
patches and reports for the problems addressed in this release.
-Because of lags created by using mirrors, this message may reach you before
-the release is available at the public sites. Sources and binaries will be
-available from the following locations:
+Sources and binaries are available from the following locations:
- - RedHat EL/CentOS 3-6 Fedora 4-14:
+ - RedHat EL/CentOS 5,6 Fedora 8,9,10,11,12,13,14:
http://prdownloads.sourceforge.net/safekeep/safekeep-common-1.3.0-1.noarch.rpm
http://prdownloads.sourceforge.net/safekeep/safekeep-client-1.3.0-1.noarch.rpm
http://prdownloads.sourceforge.net/safekeep/safekeep-server-1.3.0-1.noarch.rpm
@@ -38,14 +36,6 @@
To find out more about the project visit on our website:
http://safekeep.sourceforge.net
-You can also get the current source directly from the Subversion (SVN)
-repositories:
- http://safekeep.sourceforge.net/download.shtml#svn
-
-Patches should be submitted to "saf...@li...".
-Please don't forget to include a ChangeLog entry. If you submitted a patch,
-please check to make sure it has been included in the new release.
-
--
Dimi Paun <di...@la...>
Lattica, Inc.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 07:38:23
|
Revision: 682
http://safekeep.svn.sourceforge.net/safekeep/?rev=682&view=rev
Author: dimi
Date: 2010-11-19 07:38:15 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Tag safekeep 1.3.0
Added Paths:
-----------
safekeep/tags/Release-safekeep-1_3_0/
safekeep/tags/Release-safekeep-1_3_0/safekeep
Removed Paths:
-------------
safekeep/tags/Release-safekeep-1_3_0/safekeep
Deleted: safekeep/tags/Release-safekeep-1_3_0/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 07:32:28 UTC (rev 680)
+++ safekeep/tags/Release-safekeep-1_3_0/safekeep 2010-11-19 07:38:15 UTC (rev 682)
@@ -1,1488 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (C) 2006-2010 Lattica, Inc.
-#
-# SafeKeep is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
-#
-# Safekeep is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Safekeep. If not, see <http://www.gnu.org/licenses/>.
-
-from __future__ import generators
-import getopt, os, os.path, re, sys, fnmatch, stat
-import subprocess
-import commands, tempfile, time, traceback
-import getpass, pwd, xml.dom.minidom
-import socket, smtplib
-
-from subprocess import PIPE, STDOUT
-
-######################################################################
-# Global settings
-######################################################################
-
-config_file = '/etc/safekeep/safekeep.conf'
-config_ext = '.backup'
-trickle_cmd = 'trickle'
-logbuf = []
-is_client = False
-verbosity_level = 1
-verbosity_ssh = ''
-verbosity_trickle = ''
-work_user = getpass.getuser()
-backup_user = None
-home_dir = None
-base_dir = None
-default_bandwidth = {}
-cmd = "<Missing>"
-
-PROTOCOL = "1.1"
-VERSION = "1.3.0"
-VEBOSITY_BY_CLASS = {'DBG': 3, 'INFO': 2, 'WARN': 1, 'ERR': 0}
-
-######################################################################
-# Miscellaneous support functions
-######################################################################
-
-def send(msg):
- print msg.encode('utf-8')
- sys.stdout.flush()
-
-def log(msg, cls=None):
- global logbuf
- if cls:
- if is_client: cls = cls.lower()
- msg = '%s: %s' % (cls, msg)
- else:
- for c in VEBOSITY_BY_CLASS.keys():
- if msg.startswith(c + ': '):
- cls = c
- break
- else:
- cls = 'UNK'
-
- cutoff = VEBOSITY_BY_CLASS.get(cls.upper())
- if cutoff is None: cutoff = 3
- if is_client or verbosity_level >= cutoff:
- logbuf.append(msg)
- if is_client:
- send(msg)
- else:
- print >> sys.stderr, msg.encode('utf-8')
-
-def info_file(file, marker=None):
- info('## File: ' + file)
- errs = 0;
- fin = open(file, 'r')
- try:
- for line in fin.readlines():
- if marker:
- if line.startswith(marker):
- marker = None
- continue
- if line.startswith("Errors "):
- errs = int(line[6:])
- info(line.rstrip())
- finally:
- fin.close()
- return errs
-
-def debug(msg):
- log(msg, 'DBG')
-
-def info(msg):
- log(msg, 'INFO')
-
-def warn(msg):
- log(msg, 'WARN')
-
-def error(msg):
- log(msg, 'ERR')
-
-def args_to_list(args):
- if isinstance(args, str) or isinstance(args, unicode):
- return args.split(None)
- else:
- return args
-
-def do_spawn(args, shell=False):
- global cmd
- argslist = args_to_list(args)
- cmd = argslist[0]
- if shell:
- # If passed to a shell then give args exactly as specified.
- debug('Run [' + args + ']')
- proc = subprocess.Popen(args, bufsize=1, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- else:
- # Otherwise split into separate elements.
- debug('Run [' + ' '.join(argslist) + ']')
- proc = subprocess.Popen(argslist, bufsize=1, shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- proc.stdin.close()
- for line in proc.stdout:
- info(line.rstrip())
- proc.stdout.close()
- return proc.wait()
-
-def spawn(args, shell=False):
- global cmd
- try:
- rc = do_spawn(args, shell=shell)
- except OSError, ex:
- error("OSError: %s: %s" % (cmd, ex))
-
- if not rc:
- ret = None
- elif rc > 0:
- ret = 'exited with non zero status: %d' % rc
- elif rc < 0:
- ret = 'killed by signal: %d' % -rc
- else:
- ret = 'unknown exit status: %d' % rc
- if ret:
- error('%s failed: %s' % (cmd, ret));
- return ret
-
-def try_to_run(args, shell=False):
- try:
- rc = do_spawn(args, shell=shell)
- except OSError, ex:
- return False
- return rc in (0,1)
-
-def cmd_run(args):
- argslist = args_to_list(args)
- debug('Run [' + ' '.join(argslist) + ']')
- p = subprocess.Popen(argslist, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
- return (p.stdin, p.stdout)
-
-def send_notification(email, smtp):
- global logbuf
- if not logbuf: return
- info('Sending email to %s via %s' % (','.join(email), smtp))
- hostname = socket.gethostname()
- msg = 'From: SafeKeep@' + hostname + \
- '\r\nTo: ' + ', '.join(email) + \
- '\r\nSubject: SafeKeep results for ' + hostname + \
- '\r\n\r\n' + '\r\n'.join(logbuf)
- if smtp:
- server = smtplib.SMTP(smtp)
- server.sendmail('SafeKeep@' + hostname, email, msg)
- server.quit()
- else:
- cmd = ['/usr/sbin/sendmail', '-t']
- pin = subprocess.Popen(cmd, stdin=PIPE).stdin
- try:
- pin.write(msg)
- finally:
- pin.close()
-
-def is_temp_root(dir):
- return dir != '/'
-
-def reroot(root, path):
- if root == '/': return path
- if root.endswith('/'): root = root[:-1]
- if not path: return root
- if path.startswith('/'): return root + path
- return os.path.join(root, path)
-
-def parse_prop_file(file):
- props = {}
- fin = open(file)
- lines = fin.readlines()
- fin.close()
- for line in lines:
- line = line.strip()
- if len(line) is 0 or line[0] is '#': continue
- if '=' in line:
- key, value = line.split('=', 1)
- props[key.strip()] = value.strip()
- else:
- props[line] = None
- return props
-
-######################################################################
-# Configuration file parser
-######################################################################
-
-class ConfigException (Exception):
- def __init__(self, value):
- self.value = value
- def __str__(self):
- return repr(self.value)
-
-def parse_dump(dump_el):
- type = dump_el.getAttribute('type')
- if not type:
- raise ConfigException('You need to specify the database type')
- if type not in ('postgres', 'postgresql', 'pgsql', 'mysql'):
- raise ConfigException('Invalid database type: %s' % type)
- db = dump_el.getAttribute('db')
- user = dump_el.getAttribute('user')
- dbuser = dump_el.getAttribute('dbuser')
- dbpasswd = dump_el.getAttribute('dbpasswd')
- opts = (dump_el.getAttribute('options') or '').split()
-
- file = dump_el.getAttribute('file')
- if not file:
- raise ConfigException('You need to specify where the database should be dumped')
- cleanup = dump_el.getAttribute('cleanup')
- return { 'type' : type, 'db' : db, 'user' : user, 'dbuser' : dbuser, 'dbpasswd': dbpasswd,
- 'opts' : opts, 'file' : file, 'cleanup' : cleanup }
-
-def parse_snap(snap_el):
- device = snap_el.getAttribute('device')
- if not device:
- raise ConfigException('Please specify the device to be snapshot')
- if device.rfind('/') == -1 or device.endswith('/'):
- raise ConfigException('The device name seems incorrect: ' + device)
- size = snap_el.getAttribute('size')
- if not size:
- raise ConfigException('Please specify the size for the snapshot')
- return { 'device' : device, 'size' : size }
-
-def parse_clude(clude_el):
- path = clude_el.getAttribute('path')
- path = path.replace('*', '\*').replace('?', '\?')
- path = path.replace('[', '\[').replace(']', '\]')
- glob = clude_el.getAttribute('glob')
- regexp = clude_el.getAttribute('regexp')
- if not path and not glob and not regexp:
- raise ConfigException('Empty ' + clude_el.tagName)
- return { 'type' : clude_el.tagName, 'path' : path, 'glob' : glob, 'regexp' : regexp }
-
-def parse_bandwidth(bw_el):
- return {
- 'overall': int(bw_el.getAttribute('overall') or 0),
- 'download': int(bw_el.getAttribute('download') or 0),
- 'upload': int(bw_el.getAttribute('upload') or 0)
- }
-
-def parse_data_attributes(data_el):
- return {
- 'exclude-devices': (data_el.getAttribute('exclude-devices') or 'false'),
- 'exclude-sockets': (data_el.getAttribute('exclude-sockets') or 'false'),
- 'exclude-fifos': (data_el.getAttribute('exclude-fifos') or 'false')
- }
-
-def parse_config(backup_el, dflt_id):
- if backup_el.tagName != 'backup':
- raise ConfigException('Invalid config file, the top level element must be <backup>')
- id = backup_el.getAttribute('id')
- if not id: id = dflt_id
-
- host_el = backup_el.getElementsByTagName('host')
- if host_el:
- host = host_el[0].getAttribute('name')
- user = host_el[0].getAttribute('user')
- nice = host_el[0].getAttribute('nice')
- key_ctrl = host_el[0].getAttribute('key-ctrl')
- key_data = host_el[0].getAttribute('key-data')
- else:
- host = user = key_ctrl = key_data = None
- if host and not user:
- user = 'root'
- if host and not key_ctrl:
- key_ctrl = os.path.join('.ssh', 'safekeep-server-ctrl-key')
- if host and not key_data:
- key_data = os.path.join('.ssh', 'safekeep-server-data-key')
- if key_ctrl and not os.path.isabs(key_ctrl):
- key_ctrl = os.path.join(home_dir, key_ctrl)
- if key_data and not os.path.isabs(key_data):
- key_data = os.path.join(home_dir, key_data)
-
- bw = {}
- bw_el = backup_el.getElementsByTagName('bandwidth')
- if len(bw_el) == 1:
- bw = parse_bandwidth(bw_el[0])
- elif len(bw_el) > 1:
- raise ConfigException('Can not have more than one bandwidth element')
-
- repo_el = backup_el.getElementsByTagName('repo')
- dir = None
- retention = None
- if len(repo_el) == 1:
- dir = repo_el[0].getAttribute('path')
- retention = repo_el[0].getAttribute('retention')
- elif len(repo_el) > 1:
- raise ConfigException('Can not have more than one repo element')
- if not dir: dir = id
- dir = os.path.join(base_dir, dir)
-
- options_els = backup_el.getElementsByTagName('options')
- options = []
- if len(options_els) > 0:
- for options_el in options_els[0].childNodes:
- if options_el.nodeType != options_el.ELEMENT_NODE:
- continue
- option = options_el.nodeName
- if option == 'special-files':
- warn('options element special-files is deprecated, use data attributes instead')
- if option in ('special-files', 'rdiff-backup'):
- if options_el.hasAttributes():
- for key, value in options_el.attributes.items():
- options.append({ option : { key : value } })
- else:
- raise ConfigException('Option "%s" has no value' % option)
- else:
- raise ConfigException('Unknown option "%s"' % option)
-
- setup_el = backup_el.getElementsByTagName('setup')
- dumps = []
- snaps = []
- script = None
- if len(setup_el) > 0:
- dump_els = setup_el[0].getElementsByTagName('dump')
- for dump_el in dump_els:
- dumps.append(parse_dump(dump_el))
- snap_els = setup_el[0].getElementsByTagName('snapshot')
- for snap_el in snap_els:
- snaps.append(parse_snap(snap_el))
- script_el = setup_el[0].getElementsByTagName('script')
- if len(script_el) == 1:
- script = script_el[0].getAttribute('path')
- elif len(script_el) > 1:
- raise ConfigException('Can not have more than one setup script element')
-
- data_options = {}
- data_el = backup_el.getElementsByTagName('data')
-
- if len(data_el) == 1:
- data_options = parse_data_attributes(data_el[0])
- child_els = data_el[0].childNodes
- cludes = []
- for child_el in child_els:
- if child_el.nodeType != child_el.ELEMENT_NODE:
- continue
- if child_el.tagName not in ('include', 'exclude'):
- continue
- cludes.append(parse_clude(child_el))
- cludes.append({ 'type' : 'exclude', 'path' : '', 'glob' : '', 'regexp' : '.*' })
- elif len(data_el) > 1:
- raise ConfigException('Can not have more than one data element')
- else:
- path_xcludes = [ '/dev/', '/media/', '/mnt/', '/net/', '/proc/', '/selinux/', '/sys/',
- '/tmp/', '/var/cache', '/var/lock', '/var/run', '/var/tmp',
- '/var/named/chroot/dev', '/var/named/chroot/proc',
- '/var/named/chroot/var/run', '/var/named/chroot/var/tmp' ]
- cludes = [{ 'type' : 'exclude', 'path' : path, 'glob' : None, 'regexp' : None } for path in path_xcludes]
-
- return { 'id': id, 'host' : host, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
- 'dir' : dir, 'retention' : retention, 'dumps' : dumps, 'snaps' : snaps, 'script' : script,
- 'cludes' : cludes, 'data_options' : data_options, 'options' : options, 'bw' : bw}
-
-def parse_locs(cfglocs):
- cfgfiles = []
- for cfg in cfglocs:
- if os.path.isdir(cfg):
- for ent in os.listdir(cfg):
- if not ent.endswith(config_ext):
- warn('Ignoring file %s not ending in %s' % (os.path.join(cfg, ent), config_ext))
- continue
- filepath = os.path.join(cfg, ent)
- if not os.path.isfile(filepath):
- continue
- cfgfiles.append(filepath)
- elif os.path.isfile(cfg):
- cfgfiles.append(cfg)
- else:
- warn('Inaccessible configuration, ignoring: %s' % cfg)
-
- cfgs = {}
- for filepath in cfgfiles:
- filename = os.path.splitext(os.path.basename(filepath))[0]
-
- cfg_file = open(filepath)
- cfg_str = cfg_file.read().strip()
- cfg_file.close()
-
- dom = xml.dom.minidom.parseString(cfg_str)
- try:
- cfg = parse_config(dom.documentElement, filename)
- finally:
- dom.unlink()
- cfg['text'] = cfg_str
- if cfg['id'] in cfgs:
- raise ConfigException('Duplicate client ID: %s' % cfg['id'])
- cfgs[cfg['id']] = cfg
-
- return cfgs
-
-######################################################################
-# Script, DB and SNAPSHOT support
-# setup methods can raise exception to signal errors
-# teardown methods must succeed and cleanup the state
-######################################################################
-
-def check_script_permissions(script):
- if not os.path.isfile(script):
- return '%s is not a regular file' % script
- if not os.access(script, os.X_OK):
- return '%s is not executable' % script
-
- statinfo = os.stat(script)
- if statinfo.st_uid and statinfo.st_uid != os.getuid():
- return '%s is owned by others' % script
-
- if (statinfo.st_mode & (stat.S_IWGRP | stat.S_IWOTH)):
- return '%s is writable by others' % script
-
- return None
-
-def client_side_script(step, cfg, bdir):
- debug('Do client_side_script: step %s' % step)
-
- ret = None
- script = cfg['script']
-
- if script:
- debug('client_side_script: script = %s' % script)
- if os.path.exists(script):
- ret = check_script_permissions(script)
- if not ret:
- ret = spawn([script, step, cfg['id'], bdir])
- else:
- debug('client_side_script: %s not found' % script)
-
- return ret
-
-def do_client_dbdump(cfg):
- debug('Doing DB dumps')
- for dump in cfg['dumps']:
- type = dump['type']
- opts = dump['opts']
- passwdfile = None
- if type in ('postgres', 'postgresql', 'pgsql'):
- if dump['db']:
- args = ['pg_dump']
- args.extend(['-C'])
- else:
- args = ['pg_dumpall']
- if dump['dbuser']:
- args.extend(['-U', dump['dbuser']])
- args.extend(opts)
- if dump['db']:
- args.extend([dump['db']])
- if dump['dbpasswd']:
- (fd, passwdfile) = tempfile.mkstemp()
- f = os.fdopen(fd, 'w')
- f.write(dump['dbpasswd'])
- f.close()
-
- elif type in ('mysql'):
- args = ['mysqldump']
- if dump['dbuser']:
- args.extend(['-u', dump['dbuser']])
- if dump['dbpasswd']:
- args.extend(['-p%s' % dump['dbpasswd']])
- if not dump['db']:
- args.extend(['-A'])
- args.extend(opts)
- if dump['db']:
- args.extend([dump['db']])
-
- else:
- warn('Invalid database type: ' + type)
- continue
-
- if dump['user']:
- cmd = ' '.join([commands.mkarg(arg) for arg in args])
- args = [ 'su', '-c', cmd, '-', dump['user'] ]
- cmd = ' '.join([commands.mkarg(arg) for arg in args])
- cmd = '%s > %s' % (cmd, commands.mkarg(dump['file']))
-
-
- if passwdfile:
- os.environ['PGPASSFILE'] = passwdfile
- try:
- ec = spawn(cmd, shell=True)
- finally:
- if passwdfile:
- del os.environ['PGPASSFILE']
- os.remove(passwdfile)
- if ec:
- warn('Can not dump the database: %s' % dump['db'])
-
-def do_client_dbdump_teardown(cfg):
- debug('Tear down DB dumps')
- for dump in cfg['dumps']:
- if dump['cleanup'].lower() != 'true':
- continue
- try:
- os.remove(dump['file'])
- except Exception, e:
- warn('Unable to remove dump file: %s for database %s because: %s' %
- (dump['file'], dump['db'], e))
-
-def lvm_snap_information():
- (cin, cout) = cmd_run(['lvs', '--separator', ':', '--noheadings'])
- lines = cout.readlines()
- cout.close()
- cin.close()
- lvms = []
- for line in lines:
- if line.count(':') > 3:
- (volume, group, attr, blah1) = line.lstrip().split(':', 3)
- if fnmatch.fnmatch(volume, '*_snap_safekeep-*') and attr[0].lower() == 's':
- lvms.append([volume, group])
- return lvms
-
-def mount_information(reverse = False):
- (cin, cout) = cmd_run(['mount'])
- lines = cout.readlines()
- cout.close()
- cin.close()
- mounts = []
- pattern = re.compile(r"^(\S+) on (.+) type (\S+) \((\S+)\)")
- if reverse:
- lines.reverse()
- for line in lines:
- matches = pattern.match(line)
- if not matches is None:
- mounts.append(matches.groups())
- return mounts
-
-def map_lvm_device(device):
- device = device.replace('/mapper','').replace('-','/').replace('//', '-')
- return device.split('/')[-2:]
-
-def check_lvm_information(device):
- (group, volume) = map_lvm_device(device)
- for (lvm_volume, lvm_group) in lvm_snap_information():
- if lvm_group == group and lvm_volume.startswith(volume):
- return True
- return False
-
-def gather_lvm_information(device):
- (group, volume) = map_lvm_device(device)
- for (device, mountpoint, mounttype, mountoptions) in mount_information(False):
- if [group, volume] == map_lvm_device(device):
- return (group, volume, mountpoint, mounttype)
- return (None, None, None, None)
-
-def gather_snap_information(device, bdir):
- (group, volume, mountpoint, mounttype) = gather_lvm_information(device)
- if not mountpoint: return (None, None, None, None)
- lvmdev = os.path.join('/dev', group, volume)
- if bdir[-1] == '/': bdir = bdir[:-1]
- snapname = '%s_snap_%s' % (volume, os.path.basename(bdir))
- snapdev = os.path.join('/dev', group, snapname)
- if os.path.isabs(mountpoint[0]): mountpoint = mountpoint[1:]
- return (lvmdev, snapdev, os.path.join(bdir, mountpoint), mounttype)
-
-def do_client_snap(cfg, bdir):
- assert is_temp_root(bdir)
- debug('Doing FS snapshots')
- for snap in cfg['snaps']:
- device = snap['device']
- (lvmdev, snapdev, snapmnt, snaptyp) = gather_snap_information(device, bdir)
- if not snapmnt:
- warn('Cannot find the mountpoint for: %s' % device)
- continue
- args = ['lvcreate', '--snapshot', '--size', snap['size'],
- '--name', os.path.basename(snapdev), lvmdev]
- ec = spawn(args)
- if ec:
- warn('Can not snapshot the device: %s' % device)
- continue
- # no need to mkdir since the mountpoint already exists
- args = ['mount', '-t', snaptyp, snapdev, snapmnt]
- ec = spawn(args)
- if ec:
- warn('Can not mount the snapshot: %s' % device)
- ret = spawn(['lvremove', '--force', snapdev])
- if ret:
- warn('Can not tear down snapshot: %s' % device)
-
-def do_client_snap_teardown(cfg, bdir):
- assert is_temp_root(bdir)
- debug('Tear down FS snapshots dumps')
- snaps = list(cfg['snaps'])
- snaps.reverse()
- for snap in snaps:
- device = snap['device']
- (lvmdev, snapdev, snapmnt, snaptyp) = gather_snap_information(device, bdir)
- if not snapmnt:
- warn('Can not find the mountpoint for: %s' % device)
- continue
- ret = spawn(['umount', snapmnt])
- if ret:
- warn('Can not umount the snapshot: %s' % snapmnt)
-
- # stupid workaround for https://bugzilla.redhat.com/show_bug.cgi?id=577798
- for i in range(1,10):
- ret = spawn(['lvremove', '--force', snapdev])
- if not ret:
- break
-
- if ret:
- warn('Can not tear down snapshot: %s' % device)
-
-######################################################################
-# Client implementation
-######################################################################
-
-def do_client_config(cmd):
- cfgStr = ''
-
- (cfg_cmd, cnt_str, dflt_id) = cmd.split(':', 2)
- for i in xrange(int(cnt_str)):
- line = sys.stdin.readline()
- if not line: raise ConfigException('Unexpected end of file')
- cfgStr += line
-
- dom = xml.dom.minidom.parseString(cfgStr)
- try:
- return parse_config(dom.documentElement, dflt_id)
- finally:
- dom.unlink()
-
-def do_client_setup(cfg):
- debug('Do setup of %s' % cfg['host'])
-
- do_client_dbdump(cfg)
-
- if len(cfg['snaps']) > 0:
- debug('Checking FS snapshots')
- for snap in cfg['snaps']:
- device = snap['device']
- if check_lvm_information(device):
- raise Exception("Previous snapshots found for %s: run 'safekeep --server --cleanup' to correct" % device)
-
- ret = spawn(['modprobe', 'dm-snapshot'])
- if ret:
- warn('modprobe dm-snapshot failed, continuing')
- bdir = tempfile.mkdtemp("-rbind", "safekeep-", "/mnt")
- ret = spawn(['mount', '--rbind', '/', bdir])
- if ret:
- warn('mount --rbind failed, snapshotting will be disabled')
- try:
- os.rmdir(bdir)
- except Exception, e:
- warn('Failed to remove: %s' % bdir)
- bdir = '/'
- else:
- do_client_snap(cfg, bdir)
- else:
- bdir = '/'
- debug('Working root is %s' % bdir)
-
- return bdir
-
-def do_client_cleanup(cfg, bdir):
- debug('Do cleanup of %s in %s' % (cfg['host'], bdir))
- if is_temp_root(bdir):
- do_client_snap_teardown(cfg, bdir)
-
- ret = spawn(['umount', '-l', bdir])
- if ret:
- warn('Failed to unmount: %s' % bdir)
- else:
- try:
- os.rmdir(bdir)
- except Exception, e:
- warn('Unable to remove: ' + bdir)
-
- do_client_dbdump_teardown(cfg)
-
-def do_client_compat(server_versions):
- debug('Server versions: %s' % server_versions)
-
-def do_client_scrub():
- debug("Do client scrub loop")
-
- if os.getuid():
- if is_client:
- raise Exception('client not running as root')
- else:
- warn('--cleanup should be run as root on client')
- info('No cleanup performed')
- else:
- scrubbed = False
-
- if os.environ['PATH'][-1] == ':':
- os.environ['PATH'] += '/sbin:/usr/sbin:/usr/local/sbin:'
- else:
- os.environ['PATH'] += ':/sbin:/usr/sbin:/usr/local/sbin'
-
- # Go through and unmount anythings that are still hanging around
-
- debug("Cleaning up existing mounts")
- for (device, mountpoint, mounttype, mountoptions) in mount_information(True):
- if mountpoint.startswith('/mnt/safekeep-'):
- info("Removing mount %s" % mountpoint)
- if device == '/' and 'bind' in mountoptions.split(','):
- info("Removing rbind directory %s" % mountpoint)
- ret = spawn(['umount', '-l', mountpoint])
- if ret:
- warn('Failed to unmount: %s' % mountpoint)
- else:
- try:
- os.rmdir(mountpoint)
- except Exception, e:
- warn('Failed to remove: %s' % mountpoint)
- else:
- ret = spawn(['umount', mountpoint])
- if ret:
- warn('Can not unmount the snapshot: %s' % mountpoint)
- if fnmatch.fnmatch(device, '*_snap_safekeep-*'):
- info("Removing snapshot %s" % device)
- ret = spawn(['lvremove', '--force', device])
- if ret:
- warn('Can not tear down snapshot: %s' % device)
- scrubbed = True
-
- # Now cleanup any snapshots still hanging around
-
- debug("Cleaning up remaining snapshots")
- for (volume, group) in lvm_snap_information():
- device = os.path.join('/dev', group, volume)
- info("Removing snapshot %s" % device)
- ret = spawn(['lvremove', '--force', device])
- if ret:
- warn('Can not tear down snapshot: %s' % device)
- scrubbed = True
-
- # Now cleanup any safekeep directories still hanging around
-
- debug("Cleaning up remaining safekeep directories")
- if os.path.isdir('/mnt'):
- for ent in os.listdir('/mnt'):
- mountpoint = os.path.join('/mnt', ent)
- if ent.startswith('safekeep-') and os.path.isdir(mountpoint):
- info("Removing rbind directory %s" % mountpoint)
- try:
- os.rmdir(mountpoint)
- except Exception, e:
- warn('Failed to remove: %s' % mountpoint)
-
- if not scrubbed:
- info('No cleanup required')
-
-def do_client():
- debug("Do client main loop")
- should_cleanup = True
- bdir = '/'
- try:
- while True:
- try:
- line = sys.stdin.readline()
- if line.startswith('ALOHA'):
- do_client_compat(line.strip().split(':', 1)[1])
- send('OK %s, %s' % (PROTOCOL, VERSION))
- elif line.startswith('CONFIG'):
- cfg = do_client_config(line)
- ret = client_side_script('STARTUP', cfg, bdir)
- if ret:
- send('ERROR Client-side setup script failed: %s' % ret)
- else:
- send('OK')
- elif line.startswith('SETUP'):
- client_side_script('PRE-SETUP', cfg, bdir)
- bdir = do_client_setup(cfg)
- client_side_script('POST-SETUP', cfg, bdir)
- send('OK ' + bdir)
- elif line.startswith('CLEANUP'):
- dir = line[7:].strip()
- if dir == bdir: should_cleanup = False
- do_client_cleanup(cfg, dir)
- client_side_script('POST-BACKUP', cfg, bdir)
- send('OK')
- elif line.startswith('SCRUB'):
- do_client_scrub()
- client_side_script('POST-SCRUB', cfg, bdir)
- send('OK')
- elif not line:
- break
- else:
- send('ERROR Unknown command: %s' % line)
- break
- except Exception, e:
- traceback.print_exc(file=sys.stdout)
- send('ERROR %s' % e)
- finally:
- if should_cleanup:
- do_client_cleanup(cfg, bdir)
-
-
-######################################################################
-# Server implementation
-######################################################################
-
-def do_server_getanswer(cout):
- while True:
- line = cout.readline()
- if line.startswith('OK'):
- return line[2:-1].strip()
- elif line.startswith('ERROR'):
- raise Exception(line[5:].strip())
- elif not line:
- raise Exception('client died unexpectedly')
- else:
- log(line[:-1])
-
-def do_server_rdiff(cfg, bdir, nice, force):
- args = []
-
- if nice:
- args.extend(['nice', '-n' + str(nice)])
-
- args.extend(['rdiff-backup'])
-
- if cfg['host']:
- trickle = ''
-
- def get_bw(vals, dir):
- return vals.get(dir) or vals.get('overall')
-
- def get_bandwidth(cfg, dir):
- return get_bw(cfg['bw'], dir) or get_bw(default_bandwidth, dir)
-
- limit_dl = get_bandwidth(cfg, 'download')
- limit_ul = get_bandwidth(cfg, 'upload')
- if limit_dl or limit_ul:
- trickle = trickle_cmd + ' ' + verbosity_trickle
- if limit_dl:
- trickle += ' -d ' + str(limit_dl)
- if limit_ul:
- trickle += ' -u ' + str(limit_ul)
-
- if trickle:
- if not try_to_run(trickle_cmd + ' -V'):
- warn('Trickle not available, bandwidth limiting disabled')
- trickle = ''
-
- schema = '%s ssh %s -i %s %%s rdiff-backup --server' % (trickle, verbosity_ssh, cfg['key_data'])
- args.extend(['--remote-schema', schema])
-
- if force:
- args.extend(['--force'])
-
- options_append = []
-
- special_files = []
- if cfg['data_options'].get('exclude-devices').lower() == 'true':
- special_files.extend(['--exclude-device-files'])
- if cfg['data_options'].get('exclude-sockets').lower() == 'true':
- special_files.extend(['--exclude-sockets'])
- if cfg['data_options'].get('exclude-fifos').lower() == 'true':
- special_files.extend(['--exclude-fifos'])
-
- for option in cfg['options']:
- if 'special-files' in option:
- if 'include' in option['special-files']:
- if 'true' == option['special-files']['include'].lower():
- special_files = ['--include-special-files']
-
- # Note if we ever add other backends this section should only be run
- # when rback-diff is the current option.
-
- if 'rdiff-backup' in option:
- if 'append' in option['rdiff-backup']:
- options_append.extend(option['rdiff-backup']['append'].split(None))
-
- args.extend(special_files)
- args.extend(options_append)
-
- for clude in cfg['cludes']:
- opt = '--' + clude['type']
- if clude['path']:
- args.extend([opt, reroot(bdir, clude['path'])])
- if clude['glob']:
- args.extend([opt, reroot(bdir, clude['glob'])])
- if clude['regexp']:
- args.extend([opt + '-regexp', bdir + clude['regexp']])
-
- userhost = ''
- if cfg['host']:
- userhost = '%s@%s' % (cfg['user'], cfg['host'])
- args.extend([userhost + '::' + bdir, cfg['dir']])
- ret = spawn(args)
- if ret:
- raise Exception('Failed to run rdiff-backup')
-
-def do_server_rdiff_cleanup(cfg):
- args = ['rdiff-backup', '--check-destination-dir', cfg['dir']]
- ret = spawn(args)
- if ret:
- warn('Failed to cleanup old data, please fix the problem manually')
-
-def do_server_data_cleanup(cfg):
- args = ['rdiff-backup', '--force', '--remove-older-than', cfg['retention'], cfg['dir']]
- ret = spawn(args)
- if ret:
- warn('Failed to cleanup old data, please fix the problem manually')
-
-def do_server_compat(client_versions):
- (client_protocol, client_version) = client_versions.split(',')
- (client_major, client_minor) = client_protocol.strip().split('.')
- (server_major, server_minor) = PROTOCOL.split('.')
- if server_major != client_major:
- raise Exception('Incompatible protocols: %s <> %s' % (PROTOCOL, client_protocol))
- elif server_minor > client_minor:
- warn('Protocol mismatch: %s <> %s' % (PROTOCOL, client_protocol))
-
-def do_server(cfgs, ids, nice, force, cleanup):
- debug("Do server main loop")
- for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
- info('------------------------------------------------------------------')
- info('Server backup starting for client %s' % id)
-
- cleaned_up = 0
- try:
- if cfg['host']:
- if not os.path.isfile(cfg['key_ctrl']):
- raise Exception('Client %(id)s missing ctrl key %(key_ctrl)s' % cfg)
- if not os.path.isfile(cfg['key_data']):
- raise Exception('Client %(id)s missing data key %(id)s' % cfg)
-
- datadir = os.path.join(os.getcwd(), cfg['dir'])
- if not os.path.isdir(datadir):
- try:
- os.makedirs(datadir)
- except EnvironmentError, ex:
- raise Exception('Can not create data store dir: %s' % datadir)
-
- rdiff_logdir = os.path.join(datadir, 'rdiff-backup-data')
- if cfg['retention'] and os.path.isdir(rdiff_logdir) and not cleanup:
- do_server_data_cleanup(cfg)
-
- if cfg['host']:
- cmd = 'ssh %s -T -i %s -l %s %s safekeep --client' % (verbosity_ssh, cfg['key_ctrl'], cfg['user'], cfg['host'])
- else:
- cmd = 'safekeep --client'
- (cin, cout) = cmd_run(cmd)
-
- cin.write('ALOHA: %s, %s\n' % (PROTOCOL, VERSION))
- cin.flush()
- client_versions = do_server_getanswer(cout)
- do_server_compat(client_versions)
-
- cin.write('CONFIG: %d: %s\n' % (len(cfg['text'].splitlines()), id))
- cin.write(cfg['text'] + '\n')
- cin.flush()
- do_server_getanswer(cout)
- if cleanup:
- cin.write('SCRUB\n')
- cin.flush()
- do_server_getanswer(cout)
- bdir = '/' # Fake directory for the rest of the cleanup
- do_server_rdiff_cleanup(cfg)
- cleaned_up = 1
- errs = 0
- else:
- cin.write('SETUP\n')
- cin.flush()
- bdir = do_server_getanswer(cout)
-
- if os.path.isdir(rdiff_logdir):
- rdiff_logpre = os.listdir(rdiff_logdir)
- else:
- rdiff_logpre = []
-
- backup_log = os.path.join(rdiff_logdir, 'backup.log')
- if os.path.isfile(backup_log):
- backup_marker = '=== Backup session on %s ===' % time.asctime()
- fbm = open(backup_log, 'a')
- fbm.write(backup_marker + '\n')
- fbm.close()
- else:
- backup_marker = None
-
- do_server_rdiff(cfg, bdir, nice, force)
-
- errs = 0
- if os.path.isdir(rdiff_logdir):
- info_file(backup_log, backup_marker)
- rdiff_logpost = os.listdir(rdiff_logdir)
- for lfn in rdiff_logpost:
- if lfn.startswith('session_statistics.') and lfn.endswith('.data') and lfn not in rdiff_logpre:
- errs += info_file(os.path.join(rdiff_logdir, lfn))
- else:
- warn('Log dir does not exist.')
-
- cin.write('CLEANUP %s\n' % bdir)
- cin.flush()
- do_server_getanswer(cout)
-
- if errs == 0:
- info('Server backup for client %s: OK' % id)
- else:
- info('Server backup for client %s: OK (%d WARNINGS)' % (id, errs))
-
- except Exception, e:
- if cleanup and not cleaned_up:
- info('Client-side cleanup for client %s: FAILED' % id)
- do_server_rdiff_cleanup(cfg)
- else:
- error(e)
- error('Server backup for client %s: FAILED' % id)
-
- info('------------------------------------------------------------------')
- debug('Server backup done')
-
-def do_list(cfgs, ids, list_type, list_date, list_parsable):
- debug("Do server listing main loop")
- for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
- if list_parsable:
- info('Client: %s' % id)
- else:
- info('------------------------------------------------------------------')
- info('Server listing for client %s' % id)
-
-
- args = ['rdiff-backup']
-
- if list_type is 'increments':
- args.extend(['--list-increments'])
- elif list_type is 'sizes':
- args.extend(['--list-increment-sizes'])
- elif list_type is 'changed':
- args.extend(['--list-changed-since', list_date])
- elif list_type is 'attime':
- args.extend(['--list-at-time', list_date])
- else:
- assert False, 'Unknown list type: ' + list_type
-
- if list_parsable:
- args.extend(['--parsable-output'])
-
- args.extend([cfg['dir']])
- ret = spawn(args)
- if ret:
- raise Exception('Failed to run rdiff-backup')
-
- if not list_parsable:
- info('------------------------------------------------------------------')
- debug('Server listing done')
-
-def do_keys(cfgs, ids, nice_rem, identity, status, dump, deploy):
- for cfg in cfgs.itervalues():
- id = cfg['id']
- if ids and id not in ids: continue
- info('Handling keys for client: %s' % id)
- if not cfg['host']:
- info('%s: Client is local, it needs no keys' % id)
- continue
-
- nice = cfg['nice'] or nice_rem
- if nice:
- nice_cmd = 'nice -n%s ' % (nice)
- else:
- nice_cmd = ''
-
- cmds = ['safekeep --client', 'rdiff-backup --server --restrict-read-only /']
- privatekeyfiles = [cfg.get('key_ctrl'), cfg.get('key_data')]
- lines = []
- keys_ok = False
- for (cmd, privatekeyfile) in zip(cmds, privatekeyfiles):
- publickeyfile = privatekeyfile + '.pub'
- if not os.path.isfile(privatekeyfile):
- if os.path.isfile(publickeyfile):
- error('%s: Public key exists %s, but private key is missing. Skipping client.' % (id, publickeyfile))
- break
- if dump:
- print '%s: Key does not exist: %s.' % (id, privatekeyfile)
- break
- if status:
- print '%s: Key does not exist: %s. Will be generated.' % (id, privatekeyfile)
- break
- if deploy:
- info('%s: Key do not exist, generating it now: %s' % (id, privatekeyfile))
- gencmd = 'ssh-keygen -q -b 1024 -t dsa -N "" -C "SafeKeep auto generated key at %s@%s" -f %s' % (backup_user, os.uname()[1], privatekeyfile)
- if backup_user is not work_user:
- gencmd = 'su -s /bin/sh -c %s - %s' % (commands.mkarg(gencmd), backup_user)
- debug(gencmd)
- if subprocess.call(gencmd, shell=True):
- error('%s: Failed to generate key %s. Skipping client.' % (id, privatekeyfile))
- break
- if not os.path.isfile(publickeyfile):
- error('%s: Private key exists %s, but public key is missing. Skipping client.' % (id, privatekeyfile))
- break
- fin = open(publickeyfile, 'r')
- publickey = fin.read()
- fin.close()
- line = 'command="%s%s",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty %s' % (nice_cmd, cmd, publickey.strip())
- lines.append(line)
- else:
- keys_ok = True
-
- if not keys_ok:
- continue
-
- output = '\n'.join(lines)
- if dump:
- print output
-
- basessh = 'ssh'
- if identity: basessh += ' -i %s' % (commands.mkarg(identity))
-
- if status or deploy:
- cmd = '%s %s@%s "if test -f .ssh/authorized_keys; then cat .ssh/authorized_keys; fi"' % (basessh, cfg['user'], cfg['host'])
- debug(cmd)
- out = subprocess.Popen(cmd, shell=True, stdout=PIPE).stdout
- authtext = out.read()
- if out.close():
- warn('%s: Failed to read the authorized_keys file.' % id)
- auth_keys = parse_authorized_keys(authtext)
- this_keys = parse_authorized_keys(output)
- new_keys = []
- for this_key in this_keys:
- for auth_key in auth_keys:
- if this_key[2] == auth_key[2]: break
- else:
- new_keys.append(this_key)
- if not new_keys:
- if status:
- print '%s: Client is up to date.' % id
- continue
-
- if status:
- print '%s: Keys will be deployed on the client.' % id
- if deploy:
- cmd = '%s %s@%s "umask 077; test -d .ssh || mkdir .ssh; cat >> .ssh/authorized_keys"' % (basessh, cfg['user'], cfg['host'])
- debug(cmd)
- pipe = subprocess.Popen(cmd, shell=True, stdin=PIPE).stdin
- pipe.write('%s\n' % '\n'.join([key[4] for key in new_keys]))
- if pipe.close():
- error('Failed to deliver the keys to the client')
-
-
-# parses authozied_keys, see sshd(8) man page for details
-def parse_authorized_keys(keystext):
- keys = []
- for line in keystext.splitlines():
- line = line.strip()
- if not line or line[0] == '#': continue
- if line[0] in '0123456789':
- warn('SSH Protocol 1 keys are ignored: %s' % line)
- continue
- opts =''
- if line[0:7] not in ('ssh-dss', 'ssh-rsa'):
- in_str = False
- in_esc = False
- for i, c in enumerate(line):
- if in_str:
- if in_esc: in_esc = False
- elif c is '\'': in_esc = True
- elif c is '"': in_str = False
- else:
- if c is ' ':
- rest = line[i:].strip()
- break
- elif c is '"': in_str = True
- opts += c
- else:
- info('Invalid key line, ignoring: %s' % line)
- continue
- else:
- rest = line
-
- if rest[0] in '0123456789':
- warn('SSH Protocol 1 keys are ignored: %s' % line)
- continue
-
- parts = rest.split(None, 2)
- if len(parts) < 2:
- error('Invalid key line, skipping: %s' % line)
- continue
-
- type = parts[0]
- if type not in ('ssh-dss', 'ssh-rsa'):
- error('Invalid key type "%s", skipping: %s' % (type, line))
- continue
-
- base46enc = parts[1]
-
- if len(parts) is 2:
- comment = None
- else:
- comment = parts[2]
-
- keys.append((opts, type, base46enc, comment, line))
-
- return keys
-
-######################################################################
-# Main routine
-######################################################################
-
-def usage(exitcode=None):
- print 'usage: %s --server [common options] [server options] <client-id>*' % (sys.argv[0])
- print ' %s --keys [common options] [keys options] <client-id>*' % (sys.argv[0])
- print ' %s --list [common options] [list options] <client-id>*' % (sys.argv[0])
- print
- print 'mode selection (you must pick one):'
- print '--server launch in server mode'
- print '--keys launch in keys management mode'
- print '--list list previous backup status'
- print
- print 'common options:'
- print '-c, --conf=FILE use the FILE configuration file'
- print '-h, --help show this help message and exit'
- print '-q, --quiet decreases the verbosity level'
- print '-v, --verbose increases the verbosity level'
- print '-V, --version show the version number and exit'
- print '--noemail disables the sending of email'
- print
- print 'server options:'
- print '--force force backup destination overwriting, dangerous!'
- print '--cleanup perform cleanup actions after a failure'
- print
- print 'keys options:'
- print '-i FILE use FILE as identity for RSA/DSA authentication'
- print '--status display the key status for the clients (default)'
- print '--print display the authorization keys'
- print '--deploy deploy the authorization keys'
- print
- print 'list options:'
- print '--increments list number and dates of increments'
- print '--parsable-output tailor output for parsing by other programs'
- print '--sizes list sizes of all the increments'
- print '--changed=time list files that have changed since time'
- print '--at-time=time list files in the archive at given time'
- if exitcode is not None: sys.exit(exitcode)
-
-def main():
- try:
- opts, args = getopt.getopt(sys.argv[1:], 'c:e:i:hs:qvV',
- [ 'conf=', 'client', 'clientid=', 'deploy',
- 'email=', 'force', 'help', 'keys',
- 'list', 'increments', 'sizes',
- 'parsable-output', 'changed=', 'at-time=',
- 'noemail', 'cleanup',
- 'print', 'quiet', 'server', 'smtp=',
- 'status', 'verbose', 'version'])
- except getopt.GetoptError:
- usage(2)
-
- global backup_user, home_dir, base_dir
- mode = None
- email = []
- smtp = None
- cfgfile = None
- cfglocs = []
- verbosity = 0
- clientid = None
- force = 0
- cleanup = 0
- noemail = 0
- list_type = None
- list_parsable = 0
- list_date = None
- identity = None
- keys_status = None
- keys_print = None
- keys_deploy = None
- nice_srv = None
- for o, a in opts:
- if o in ('-c', '--conf'):
- if os.path.isdir(a) or a.endswith(config_ext):
- warn('Adding client config files/dirs via this switch is deprecated')
- cfglocs.append(a)
- elif cfgfile is None:
- cfgfile = a
- else:
- error('A main configuration file can be specified only once!')
- sys.exit(2)
- elif o in ('-e', '--email'):
- warn('The -e/--email options are deprecated and will be removed in the future')
- warn('Please use the /etc/safekeep/safekeep.conf file instead')
- email.append(a)
- elif o in ('-h', '--help'):
- usage(0)
- elif o in ('-s', '--smtp'):
- warn('The -s/--smtp options are deprecated and will be removed in the future')
- warn('Please use the /etc/safekeep/safekeep.conf file instead')
- smtp = a
- elif o in ('--server', ):
- if mode: usage(2)
- mode = 'server'
- elif o in ('--list', ):
- if mode: usage(2)
- mode = 'list'
- elif o in ('--client', ):
- if mode: usage(2)
- mode = 'client'
- elif o in ('--keys', ):
- if mode: usage(2)
- mode = 'keys'
- elif o in ('--force', ):
- force = 1
- elif o in ('--cleanup', ):
- cleanup = 1
- elif o in ('--noemail', ):
- noemail = 1
- elif o in ('--increments', ):
- if list_type: usage(2)
- list_type = 'increments'
- elif o in ('--sizes', ):
- if list_type: usage(2)
- list_type = 'sizes'
- elif o in ('--parsable-output', ):
- list_parsable = 1
- elif o in ('--changed', ):
- if list_type: usage(2)
- list_type = 'changed'
- list_date = a
- elif o in ('--at-time', ):
- if list_type: usage(2)
- list_type = 'attime'
- list_date = a
- elif o in ('-i', ):
- identity = a
- elif o in ('--status', ):
- keys_status = True
- elif o in ('--print', ):
- keys_print = True
- elif o in ('--deploy', ):
- keys_deploy = True
- elif o in ('-q', '--quiet'):
- verbosity -= 1
- elif o in ('-v', '--verbose'):
- verbosity += 1
- elif o in ('-V', '--version'):
- print 'safekeep', VERSION
- return
-
- if mode is None:
- usage(2)
-
- if mode is not 'keys' and (identity or keys_status or keys_print or keys_deploy):
- usage(2)
-
- if mode is not 'list' and (list_type or list_date or list_parsable):
- usage(2)
-
- if mode is not 'server' and (email or smtp):
- usage(2)
-
- if not mode in ['server', 'client'] and cleanup:
- usage(2)
-
- if mode is 'client' and cfglocs:
- usage(2)
-
- if mode is not 'client':
- if cfgfile is None and os.path.isfile(config_file):
- cfgfile = config_file
- if cfgfile and os.path.isfile(cfgfile):
- props = parse_prop_file(cfgfile)
- else:
- if cfgfile:
- warn('Configuration file does not exist, skipping: %s' % cfgfile)
- else:
- cfgfile = config_file
- props = {}
-
- def get_int(p):
- v = props.get(p)
- if v is not None and v is not '':
- return int(v)
- return None
-
- if 'backup.user' in props:
- backup_user = props['backup.user']
- if 'base.dir' in props:
- base_dir = props['base.dir']
- if 'email.smtp.server' in props:
- smtp = props['email.smtp.server']
- if 'email.to' in props:
- email = props['email.to'].split(',')
- nice_def = get_int('nice.adjustment')
- nice_srv = get_int('nice.adjustment.server') or nice_def
- nice_cln = get_int('nice.adjustment.client') or nice_def
-
- global default_bandwidth
- default_bandwidth['overall'] = get_int('bandwidth.limit') or 0
- default_bandwidth['download'] = get_int('bandwidth.limit.download') or 0
- default_bandwidth['upload'] = get_int('bandwidth.limit.upload') or 0
-
- if len(cfglocs) == 0:
- locs = os.path.join(os.path.dirname(cfgfile), 'backup.d')
- if os.path.isdir(locs): cfglocs.append(locs)
-
- if backup_user and backup_user != work_user:
- (user, pswd, uid, gid, gecos, home_dir, shell) = pwd.getpwnam(backup_user)
- if mode is not 'keys':
- os.setregid(gid, gid)
- os.setreuid(uid, uid)
- os.environ['HOME'] = home_dir
- else:
- backup_user = work_user
- home_dir = os.getenv('HOME', '/')
-
- if not base_dir:
- base_dir = home_dir
-
- if len(cfglocs) > 0:
- cfgs = parse_locs(cfglocs)
- else:
- cfgs = {}
-
- if mode is 'client':
- if len(args) > 0: usage(2)
- else:
- ok = True
- for arg in args:
- if arg in cfgs: continue
- error('Unknown client ID: %s' % arg)
- if os.path.isfile(arg):
- error('It appears to be a file, configuration files are passed via the -c/--conf switch.')
- ok = False
- if not ok: sys.exit(2)
-
- try:
- global is_client, verbosity_level, verbosity_ssh, verbosity_trickle
-
- if verbosity > 2:
- verbosity_trickle = verbosity_ssh = '-' + (verbosity-2) * 'v'
- if mode is 'server':
- is_client = False
- verbosity_level = 1 + verbosity
- do_server(cfgs, args, nice_srv, force, cleanup)
- elif mode is 'list':
- if list_type is None:
- list_type = 'increments'
- is_client = False
- verbosity_level = 2 + verbosity
- do_list(cfgs, args, list_type, list_date, list_parsable)
- elif mode is 'client':
- if cleanup:
- is_client = False
- verbosity_level = 1 + verbosity
- do_client_scrub()
- else:
- is_client = True
- verbosity_level = 3 + verbosity
- do_client()
- elif mode is 'keys':
- is_client = False
- verbosity_level = 1 + verbosity
- if not keys_status and not keys_print and not keys_deploy:
- keys_status = True
- do_keys(cfgs, args, nice_cln, identity, keys_status, keys_print, keys_deploy)
- else:
- assert False, 'Unknown mode: %s' % (mode)
- except Exception, ex:
- traceback.print_exc(file=sys.stdout)
- error('ERROR: %s' % ex)
-
- if email and not noemail:
- send_notification(email, smtp)
-
-if __name__ == '__main__':
- main()
-
-# vim: et ts=8 sw=4 sts=4
Copied: safekeep/tags/Release-safekeep-1_3_0/safekeep (from rev 681, safekeep/trunk/safekeep)
===================================================================
--- safekeep/tags/Release-safekeep-1_3_0/safekeep (rev 0)
+++ safekeep/tags/Release-safekeep-1_3_0/safekeep 2010-11-19 07:38:15 UTC (rev 682)
@@ -0,0 +1,1488 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2006-2010 Lattica, Inc.
+#
+# SafeKeep is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# Safekeep is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Safekeep. If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import generators
+import getopt, os, os.path, re, sys, fnmatch, stat
+import subprocess
+import commands, tempfile, time, traceback
+import getpass, pwd, xml.dom.minidom
+import socket, smtplib
+
+from subprocess import PIPE, STDOUT
+
+######################################################################
+# Global settings
+######################################################################
+
+config_file = '/etc/safekeep/safekeep.conf'
+config_ext = '.backup'
+trickle_cmd = 'trickle'
+logbuf = []
+is_client = False
+verbosity_level = 1
+verbosity_ssh = ''
+verbosity_trickle = ''
+work_user = getpass.getuser()
+backup_user = None
+home_dir = None
+base_dir = None
+default_bandwidth = {}
+cmd = "<Missing>"
+
+PROTOCOL = "1.1"
+VERSION = "1.3.0"
+VEBOSITY_BY_CLASS = {'DBG': 3, 'INFO': 2, 'WARN': 1, 'ERR': 0}
+
+######################################################################
+# Miscellaneous support functions
+######################################################################
+
+def send(msg):
+ print msg.encode('utf-8')
+ sys.stdout.flush()
+
+def log(msg, cls=None):
+ global logbuf
+ if cls:
+ if is_client: cls = cls.lower()
+ msg = '%s: %s' % (cls, msg)
+ else:
+ for c in VEBOSITY_BY_CLASS.keys():
+ if msg.startswith(c + ': '):
+ cls = c
+ break
+ else:
+ cls = 'UNK'
+
+ cutoff = VEBOSITY_BY_CLASS.get(cls.upper())
+ if cutoff is None: cutoff = 3
+ if is_client or verbosity_level >= cutoff:
+ logbuf.append(msg)
+ if is_client:
+ send(msg)
+ else:
+ print >> sys.stderr, msg.encode('utf-8')
+
+def info_file(file, marker=None):
+ info('## File: ' + file)
+ errs = 0;
+ fin = open(file, 'r')
+ try:
+ for line in fin.readlines():
+ if marker:
+ if line.startswith(marker):
+ marker = None
+ continue
+ if line.startswith("Errors "):
+ errs = int(line[6:])
+ info(line.rstrip())
+ finally:
+ fin.close()
+ return errs
+
+def debug(msg):
+ log(msg, 'DBG')
+
+def info(msg):
+ log(msg, 'INFO')
+
+def warn(msg):
+ log(msg, 'WARN')
+
+def error(msg):
+ log(msg, 'ERR')
+
+def args_to_list(args):
+ if isinstance(args, str) or isinstance(args, unicode):
+ return args.split(None)
+ else:
+ return args
+
+def do_spawn(args, shell=False):
+ global cmd
+ argslist = args_to_list(args)
+ cmd = argslist[0]
+ if shell:
+ # If passed to a shell then give args exactly as specified.
+ debug('Run [' + args + ']')
+ proc = subprocess.Popen(args, bufsize=1, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ else:
+ # Otherwise split into separate elements.
+ debug('Run [' + ' '.join(argslist) + ']')
+ proc = subprocess.Popen(argslist, bufsize=1, shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
+ proc.stdin.close()
+ for line in proc.stdout:
+ info(line.rstrip())
+ proc.stdout.close()
+ return proc.wait()
+
+def spawn(args, shell=False):
+ global cmd
+ try:
+ rc = do_spawn(args, shell=shell)
+ except OSError, ex:
+ error("OSError: %s: %s" % (cmd, ex))
+
+ if not rc:
+ ret = None
+ elif rc > 0:
+ ret = 'exited with non zero status: %d' % rc
+ elif rc < 0:
+ ret = 'killed by signal: %d' % -rc
+ else:
+ ret = 'unknown exit status: %d' % rc
+ if ret:
+ error('%s failed: %s' % (cmd, ret...
[truncated message content] |
|
From: <di...@us...> - 2010-11-19 07:34:18
|
Revision: 681
http://safekeep.svn.sourceforge.net/safekeep/?rev=681&view=rev
Author: dimi
Date: 2010-11-19 07:34:12 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Make sure we init the var
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 07:32:28 UTC (rev 680)
+++ safekeep/trunk/safekeep 2010-11-19 07:34:12 UTC (rev 681)
@@ -286,7 +286,7 @@
key_ctrl = host_el[0].getAttribute('key-ctrl')
key_data = host_el[0].getAttribute('key-data')
else:
- host = user = key_ctrl = key_data = None
+ host = user = nice = key_ctrl = key_data = None
if host and not user:
user = 'root'
if host and not key_ctrl:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 07:32:35
|
Revision: 680
http://safekeep.svn.sourceforge.net/safekeep/?rev=680&view=rev
Author: dimi
Date: 2010-11-19 07:32:28 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Prep release 1.3.0
Modified Paths:
--------------
safekeep/trunk/ANNOUNCE
Modified: safekeep/trunk/ANNOUNCE
===================================================================
--- safekeep/trunk/ANNOUNCE 2010-11-19 07:24:53 UTC (rev 679)
+++ safekeep/trunk/ANNOUNCE 2010-11-19 07:32:28 UTC (rev 680)
@@ -1,30 +1,39 @@
-This is release 1.2.1 of SafeKeep, a centralized and easy to use
+This is release 1.3.0 of SafeKeep, a centralized and easy to use
backup application that combines the best features of a mirror
and an incremental backup.
What's new in this release:
- - Fix error while trying to nice the server
+ - Allow control of nice value on both server and client
+ - Fix password handling when dumping MySQL databases.
+ - Fix MySQL dumps when passing a username (for newer versions).
+ - Handle correctly Unicode strings, such as localized DB names.
+ - Fix a bug when dumping a specific Postgresql database.
+ - Prepare the code for newer Python versions
+ - Drop support for Python < 2.4, subprocess module isn't supported.
+ - Try to remove a snapshot up to 10 times in a row to workaround
+ silly udev bug: https://bugzilla.redhat.com/show_bug.cgi?id=577798
+ - Multiple internal cleanups, and minor bugs fixes.
-Thanks go to Bryan Talbot and Frank Crawford for providing patches
-for the problems addressed in this release.
+Thanks go to Frank Crawford and Bertrand Lecervoisier for providing
+patches and reports for the problems addressed in this release.
Because of lags created by using mirrors, this message may reach you before
the release is available at the public sites. Sources and binaries will be
available from the following locations:
- - RedHat EL 3,4,5, CentOS 3,4,5, Fedora 4,5,6,7,8,9,10:
- http://prdownloads.sourceforge.net/safekeep/safekeep-common-1.2.1-1.noarch.rpm
- http://prdownloads.sourceforge.net/safekeep/safekeep-client-1.2.1-1.noarch.rpm
- http://prdownloads.sourceforge.net/safekeep/safekeep-server-1.2.1-1.noarch.rpm
- http://prdownloads.sourceforge.net/safekeep/safekeep-1.2.1-1.src.rpm
+ - RedHat EL/CentOS 3-6 Fedora 4-14:
+ http://prdownloads.sourceforge.net/safekeep/safekeep-common-1.3.0-1.noarch.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-client-1.3.0-1.noarch.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-server-1.3.0-1.noarch.rpm
+ http://prdownloads.sourceforge.net/safekeep/safekeep-1.3.0-1.src.rpm
- - Ubuntu Edgy, Dapper, and Breezy:
- http://prdownloads.sourceforge.net/safekeep/safekeep-common_1.2.1_all.deb
- http://prdownloads.sourceforge.net/safekeep/safekeep-client_1.2.1_all.deb
- http://prdownloads.sourceforge.net/safekeep/safekeep-server_1.2.1_all.deb
+ - Ubuntu Edgy, Dapper, Breezy, Hardy, Karmic, Lucid, Maverick, and Natty:
+ http://prdownloads.sourceforge.net/safekeep/safekeep-common_1.3.0_all.deb
+ http://prdownloads.sourceforge.net/safekeep/safekeep-client_1.3.0_all.deb
+ http://prdownloads.sourceforge.net/safekeep/safekeep-server_1.3.0_all.deb
- Source:
- http://prdownloads.sourceforge.net/safekeep/safekeep-1.2.1.tar.gz
+ http://prdownloads.sourceforge.net/safekeep/safekeep-1.3.0.tar.gz
To find out more about the project visit on our website:
http://safekeep.sourceforge.net
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 07:24:59
|
Revision: 679
http://safekeep.svn.sourceforge.net/safekeep/?rev=679&view=rev
Author: dimi
Date: 2010-11-19 07:24:53 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Work around for udev bug https://bugzilla.redhat.com/show_bug.cgi?id=577798
Modified Paths:
--------------
safekeep/trunk/safekeep
safekeep/trunk/safekeep.spec.in
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 07:14:31 UTC (rev 678)
+++ safekeep/trunk/safekeep 2010-11-19 07:24:53 UTC (rev 679)
@@ -615,7 +615,13 @@
ret = spawn(['umount', snapmnt])
if ret:
warn('Can not umount the snapshot: %s' % snapmnt)
- ret = spawn(['lvremove', '--force', snapdev])
+
+ # stupid workaround for https://bugzilla.redhat.com/show_bug.cgi?id=577798
+ for i in range(1,10):
+ ret = spawn(['lvremove', '--force', snapdev])
+ if not ret:
+ break
+
if ret:
warn('Can not tear down snapshot: %s' % device)
Modified: safekeep/trunk/safekeep.spec.in
===================================================================
--- safekeep/trunk/safekeep.spec.in 2010-11-19 07:14:31 UTC (rev 678)
+++ safekeep/trunk/safekeep.spec.in 2010-11-19 07:24:53 UTC (rev 679)
@@ -124,6 +124,8 @@
- Fix a bug when dumping a specific Postgresql database.
- Prepare the code for newer Python versions
- Drop support for Python < 2.4, subprocess module isn't supported.
+ - Try to remove a snapshot up to 10 times in a row to workaround
+ silly udev bug: https://bugzilla.redhat.com/show_bug.cgi?id=577798
- Multiple internal cleanups, and minor bugs fixes.
* Thu Apr 30 2009 Dimi Paun <di...@la...> 1.2.1-1
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 07:14:37
|
Revision: 678
http://safekeep.svn.sourceforge.net/safekeep/?rev=678&view=rev
Author: dimi
Date: 2010-11-19 07:14:31 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Prep release
Modified Paths:
--------------
safekeep/trunk/safekeep
safekeep/trunk/safekeep.spec.in
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 07:02:19 UTC (rev 677)
+++ safekeep/trunk/safekeep 2010-11-19 07:14:31 UTC (rev 678)
@@ -1,6 +1,6 @@
#!/usr/bin/env python
-# Copyright (C) 2006-2007 Lattica, Inc.
+# Copyright (C) 2006-2010 Lattica, Inc.
#
# SafeKeep is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -44,7 +44,7 @@
cmd = "<Missing>"
PROTOCOL = "1.1"
-VERSION = "1.2.1"
+VERSION = "1.3.0"
VEBOSITY_BY_CLASS = {'DBG': 3, 'INFO': 2, 'WARN': 1, 'ERR': 0}
######################################################################
Modified: safekeep/trunk/safekeep.spec.in
===================================================================
--- safekeep/trunk/safekeep.spec.in 2010-11-19 07:02:19 UTC (rev 677)
+++ safekeep/trunk/safekeep.spec.in 2010-11-19 07:14:31 UTC (rev 678)
@@ -24,7 +24,7 @@
Summary: The SafeKeep backup system (common component)
Group: Applications/System
Requires: rdiff-backup
-Requires: python >= 2.2
+Requires: python >= 2.4
%description common
SafeKeep is a client/server backup system which enhances the
@@ -116,6 +116,16 @@
%doc samples/sample.backup
%changelog
+* Fri Nov 19 2010 Dimi Paun <di...@la...> 1.3.0-1
+ - Allow control of nice value on both server and client
+ - Fix password handling when dumping MySQL databases.
+ - Fix MySQL dumps when passing a username (for newer versions).
+ - Handle correctly Unicode strings, such as localized DB names.
+ - Fix a bug when dumping a specific Postgresql database.
+ - Prepare the code for newer Python versions
+ - Drop support for Python < 2.4, subprocess module isn't supported.
+ - Multiple internal cleanups, and minor bugs fixes.
+
* Thu Apr 30 2009 Dimi Paun <di...@la...> 1.2.1-1
- Fix error while trying to nice the server
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 07:02:25
|
Revision: 677
http://safekeep.svn.sourceforge.net/safekeep/?rev=677&view=rev
Author: dimi
Date: 2010-11-19 07:02:19 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Oops, typo
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 06:55:25 UTC (rev 676)
+++ safekeep/trunk/safekeep 2010-11-19 07:02:19 UTC (rev 677)
@@ -1072,7 +1072,7 @@
nice = cfg['nice'] or nice_rem
if nice:
- nice_cmd = 'nice -n%s' % (nice)
+ nice_cmd = 'nice -n%s ' % (nice)
else:
nice_cmd = ''
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 06:55:31
|
Revision: 676
http://safekeep.svn.sourceforge.net/safekeep/?rev=676&view=rev
Author: dimi
Date: 2010-11-19 06:55:25 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Add precise control of nice values for both server and
client side of things. By default, safekeep will run now
as nice +10 on both sides.
However, existing installs will have to redeploy keys to
get this running on the client side.
Modified Paths:
--------------
safekeep/trunk/doc/safekeep.backup.txt
safekeep/trunk/doc/safekeep.conf.txt
safekeep/trunk/safekeep
safekeep/trunk/safekeep.conf
safekeep/trunk/samples/sample.backup
Modified: safekeep/trunk/doc/safekeep.backup.txt
===================================================================
--- safekeep/trunk/doc/safekeep.backup.txt 2010-11-19 05:58:45 UTC (rev 675)
+++ safekeep/trunk/doc/safekeep.backup.txt 2010-11-19 06:55:25 UTC (rev 676)
@@ -25,7 +25,7 @@
<!-- the client backup host, the user under which the servers will connect,
the SSH keys used for control and data transfer -->
<host
- name="myhost" user="root"
+ name="myhost" user="root" nice="10"
key-ctrl="/home/jdoe/.ssh/backup_id_dsa"
key-data="/home/jdoe/.ssh/backup2_id_dsa"
/>
@@ -113,6 +113,12 @@
so it most likely needs to be 'root'.
Optional, defaults to 'root'.
+/backup/host/@nice::
+ The nice adjustment for the client. This settings is normally
+ not all that important, as most of the load rests on the server side.
+ NB: if you change this value, you will have to re-deploy the auth keys.
+ Optional, defaults to no nice level.
+
/backup/host/@key-ctrl::
This is the private key used to establish the SSH connection
to the client for the control channel. Use of the default value
Modified: safekeep/trunk/doc/safekeep.conf.txt
===================================================================
--- safekeep/trunk/doc/safekeep.conf.txt 2010-11-19 05:58:45 UTC (rev 675)
+++ safekeep/trunk/doc/safekeep.conf.txt 2010-11-19 06:55:25 UTC (rev 676)
@@ -46,13 +46,23 @@
`/usr/sbin/sendmail` to deliver the mail.
nice.adjustment::
- The nice level adjustment for safekeep, for the time
- being used only on the server side.
+ The default nice level adjustment for safekeep.
It specifies an integer to be added to the current nice
level. Nicenesses range from -20 (most favorable scheduling)
to 19 (least favorable).
If no nice level is specified, safekeep is not niced.
+nice.adjustment.server::
+ The nice level adjustment for safekeep, used on the server side.
+ It overrides the generic setting in nice.adjustment.
+
+nice.adjustment.client::
+ The default nice adjustment for the client. This settings is normally
+ not all that important, as most of the load is on the server side.
+ You can also set the remove nice level on a per-client basis in
+ the .backup file (see /backup/host/@nice).
+ NB: if you change this value, you will have to re-deploy the auth keys.
+
bandwidth.overall::
This is the default bandwidth limit for both upload and
download for all the clients. It is an integer number of KB/s
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-11-19 05:58:45 UTC (rev 675)
+++ safekeep/trunk/safekeep 2010-11-19 06:55:25 UTC (rev 676)
@@ -282,6 +282,7 @@
if host_el:
host = host_el[0].getAttribute('name')
user = host_el[0].getAttribute('user')
+ nice = host_el[0].getAttribute('nice')
key_ctrl = host_el[0].getAttribute('key-ctrl')
key_data = host_el[0].getAttribute('key-data')
else:
@@ -373,7 +374,7 @@
'/var/named/chroot/var/run', '/var/named/chroot/var/tmp' ]
cludes = [{ 'type' : 'exclude', 'path' : path, 'glob' : None, 'regexp' : None } for path in path_xcludes]
- return { 'id': id, 'host' : host, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
+ return { 'id': id, 'host' : host, 'nice' : nice, 'user' : user, 'key_ctrl' : key_ctrl, 'key_data' : key_data,
'dir' : dir, 'retention' : retention, 'dumps' : dumps, 'snaps' : snaps, 'script' : script,
'cludes' : cludes, 'data_options' : data_options, 'options' : options, 'bw' : bw}
@@ -1060,7 +1061,7 @@
info('------------------------------------------------------------------')
debug('Server listing done')
-def do_keys(cfgs, ids, identity, status, dump, deploy):
+def do_keys(cfgs, ids, nice_rem, identity, status, dump, deploy):
for cfg in cfgs.itervalues():
id = cfg['id']
if ids and id not in ids: continue
@@ -1068,6 +1069,13 @@
if not cfg['host']:
info('%s: Client is local, it needs no keys' % id)
continue
+
+ nice = cfg['nice'] or nice_rem
+ if nice:
+ nice_cmd = 'nice -n%s' % (nice)
+ else:
+ nice_cmd = ''
+
cmds = ['safekeep --client', 'rdiff-backup --server --restrict-read-only /']
privatekeyfiles = [cfg.get('key_ctrl'), cfg.get('key_data')]
lines = []
@@ -1099,7 +1107,7 @@
fin = open(publickeyfile, 'r')
publickey = fin.read()
fin.close()
- line = 'command="%s",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty %s' % (cmd, publickey.strip())
+ line = 'command="%s%s",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty %s' % (nice_cmd, cmd, publickey.strip())
lines.append(line)
else:
keys_ok = True
@@ -1386,7 +1394,9 @@
smtp = props['email.smtp.server']
if 'email.to' in props:
email = props['email.to'].split(',')
- nice_srv = get_int('nice.adjustment')
+ nice_def = get_int('nice.adjustment')
+ nice_srv = get_int('nice.adjustment.server') or nice_def
+ nice_cln = get_int('nice.adjustment.client') or nice_def
global default_bandwidth
default_bandwidth['overall'] = get_int('bandwidth.limit') or 0
@@ -1456,7 +1466,7 @@
verbosity_level = 1 + verbosity
if not keys_status and not keys_print and not keys_deploy:
keys_status = True
- do_keys(cfgs, args, identity, keys_status, keys_print, keys_deploy)
+ do_keys(cfgs, args, nice_cln, identity, keys_status, keys_print, keys_deploy)
else:
assert False, 'Unknown mode: %s' % (mode)
except Exception, ex:
Modified: safekeep/trunk/safekeep.conf
===================================================================
--- safekeep/trunk/safekeep.conf 2010-11-19 05:58:45 UTC (rev 675)
+++ safekeep/trunk/safekeep.conf 2010-11-19 06:55:25 UTC (rev 676)
@@ -10,7 +10,10 @@
# the base directory for data repository relative paths
base.dir = /var/lib/safekeep
-# by default, be nice to the server during backup
+# by default, be nice during backup
+# you can control the server/client nice level via
+# nice.adjustment.server and nice.adjustment.client
+# or even on a per-client box in the .backup file.
nice.adjustment = 10
# A default bandwidth limit for both download/upload
Modified: safekeep/trunk/samples/sample.backup
===================================================================
--- safekeep/trunk/samples/sample.backup 2010-11-19 05:58:45 UTC (rev 675)
+++ safekeep/trunk/samples/sample.backup 2010-11-19 06:55:25 UTC (rev 676)
@@ -4,7 +4,7 @@
<!-- the client backup host, the user under which the servers will connect,
the SSH keys used for launching "safekeep -c" and "rdiff-backup" -->
<host
- name="my_box.corp.com" user="root"
+ name="my_box.corp.com" user="root" nice="10"
key-ctrl="/home/jdoe/.ssh/backup_id_dsa"
key-data="/home/jdoe/.ssh/backup2_id_dsa"
/>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-11-19 05:58:51
|
Revision: 675
http://safekeep.svn.sourceforge.net/safekeep/?rev=675&view=rev
Author: dimi
Date: 2010-11-19 05:58:45 +0000 (Fri, 19 Nov 2010)
Log Message:
-----------
Ack Frank, he's done a lot of good work on this one
Modified Paths:
--------------
safekeep/trunk/AUTHORS
Modified: safekeep/trunk/AUTHORS
===================================================================
--- safekeep/trunk/AUTHORS 2010-02-10 16:06:37 UTC (rev 674)
+++ safekeep/trunk/AUTHORS 2010-11-19 05:58:45 UTC (rev 675)
@@ -1,6 +1,7 @@
SafeKeep is available thanks to the work of:
Dimi Paun <di...@la...>
Stelian Pop <st...@la...>
+ Frank Crawford <fr...@cr...>
Special thanks for testing, ideas, and support go to:
Bogdan Mincic <bo...@la...>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <di...@us...> - 2010-02-10 16:06:43
|
Revision: 674
http://safekeep.svn.sourceforge.net/safekeep/?rev=674&view=rev
Author: dimi
Date: 2010-02-10 16:06:37 +0000 (Wed, 10 Feb 2010)
Log Message:
-----------
Bertrand Lecervoisier <ber...@la...>
Use the correct package name.
Modified Paths:
--------------
safekeep/trunk/safekeep
Modified: safekeep/trunk/safekeep
===================================================================
--- safekeep/trunk/safekeep 2010-02-09 16:01:11 UTC (rev 673)
+++ safekeep/trunk/safekeep 2010-02-10 16:06:37 UTC (rev 674)
@@ -499,7 +499,7 @@
if passwdfile:
- environ['PGPASSFILE'] = passwdfile
+ os.environ['PGPASSFILE'] = passwdfile
try:
ec = spawn(cmd, shell=True)
finally:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
