Rsync Vault Manager

rvm 1.05 file released: rvm_1.05.tar.gz

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-README

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm_1.05_i386.deb

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm_1.05.dsc

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-Changelog

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-bin-i386.tgz

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-bin-amd64.tgz

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm_1.05_amd64.deb

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-2.x86_64.rpm

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

rvm 1.05 file released: rvm-1.05-2.i386.rpm

rvm (1.05) intrepid; urgency=low * Update: Updates to code to compile w/ gcc 4.3.2 * Bug: In test-rmath, absolute value of a signed char -128 == -128 * Bugfix: Depending on the situation, test_cwd() in test-fs.cc will always fail, even though it's not a bug. This test has been removed. * Bugfix: job_archiver::mf_process_rsync_io() had flawed rsync-timeout logic. * Bugfix: When rsync-timout event occurs, job was treated as though it had succeeded b/c rsync didn't report an error code or a signal when it was killed. Now rsync-timeout sets a flag that is checked, in addition to signal and exit code returned by rsync, when evalulating whether or not the job was successful. -- Michael Peek <peek@home.nimbios.org> Fri, 12 Jun 2009 10:00:21 -0400 rvm (1.04b) stable; urgency=low * Bugfix: job-path processing was ignoring rsync_behavior::ok setting. * Added new rsync exit codes to man page and exit-code strings. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 28 Aug 2008 09:40:43 -0400 rvm (1.04a) stable; urgency=low * Bugfix: Report generation should take rsync-behavior into account in addition to rsync exit code and signal generated when deciding whether or not a job path was OK or ERROR. Bugfix needs testing before promoting to 1.05 -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 27 Aug 2008 12:34:38 -0400 rvm (1.04) stable; urgency=low * Bugfix: rconfig's job::generate_rsync_options_vector() has a completely new parser. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Wed, 2 Jul 2008 13:05:57 -0400 rvm (1.03) stable; urgency=low * Bugfix: Applied patch submitted by Ezra Peisach to fix a bug in rsync-options parsing that would result in an empty argument string being passed to rsync by mistake. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Tue, 1 Jul 2008 12:01:59 -0400 rvm (1.02) stable; urgency=low * Feature: Added rsync-retry-delay to specify how long to wait (in minutes) before rerunning rsync after a failed attempt. In the event that rsync failed because of an open and changing file that it could not read the last time, this will give the user or OS time to finish and close that file. (A better solution would be to watch the file and back it up as soon as it's free, but that is outside the scope of rvm.) * Feature: Added rsync-multi-hardlink option for people with newer versions of rsync that can take advantage of using multiple --link-dest=<path> command line options to rsync. * Feature: Added rsync-multi-hardlink-max option to specify a maximum number of --link-dest=<path> options to pass to rsync. Currently the maximum that rsync will accept is 20, so that is used as the default. Rvm will accept higher values, but higher values will cause rsync to exit with an error. (Not enforcing a maximum in rvm allows for future changes in rsync, but allows for user error.) * Bugfix: A simple bug in implementation caused rsync-retry-count to be off by one. Setting rsync-retry-count to 0 would cause rsync never to run at all, and a default setting of 3 would mean that rsync is only retried 2 times. * After a system upgrade to Debian/Etch, the first order of business is to make sure that the latest version of RVM still compiles and passes all of it's tests. - The following error message was produced during compilation: tortoise% make cd . && /bin/sh /home/peek/prog/rvm/src/missing --run automake-1.10 --foreign --include-deps Makefile Makefile.am:1090: `#' comment at start of rule is unportable Makefile.am:1091: `#' comment at start of rule is unportable Makefile.am:1092: `#' comment at start of rule is unportable Makefile.am:1095: `#' comment at start of rule is unportable Makefile.am:1096: `#' comment at start of rule is unportable Makefile.am:1097: `#' comment at start of rule is unportable Makefile.am:1100: `#' comment at start of rule is unportable Makefile.am:1101: `#' comment at start of rule is unportable Makefile.am:1102: `#' comment at start of rule is unportable Makefile.am:1116: `#' comment at start of rule is unportable make: *** [Makefile.in] Error 1 Removed '#' comments from Makefile.am to remove these errors. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT estring.o -MD -MP -MF .deps/estring.Tpo -c -o estring.o estring.cc estring.h:259: error: extra qualification estring:: on member T_fraction_to_strings estring.h:262: error: extra qualification estring:: on member T_integral_to_string estring.h:265: error: extra qualification estring:: on member T_string_to_integral estring.h:269: error: extra qualification estring:: on member T_string_to_signed_integral estring.h:272: error: extra qualification estring:: on member T_string_to_fractional estring.cc: In member function void estring::T_integral_to_string(const T&, std::string&) [with T = short unsigned int]: estring.cc:1130: instantiated from here estring.cc:65: warning: comparison is always true due to limited range of data type make[1]: *** [estring.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "estring::" from before T_* templated functions. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:269: error: using typedef-name configuration_manager::cfgfiles_type after class rconfig.h:225: error: configuration_manager::cfgfiles_type has a previous declaration here rconfig.h:369: error: extra qualification job_parser:: on member read_job rconfig.cc: In member function void configuration_manager::clear(): rconfig.cc:1004: error: request for member clear in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc: In member function void configuration_manager::init(int, char**): rconfig.cc:1111: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1149: error: request for member push_back in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member begin in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int rconfig.cc:1178: error: request for member end in ((configuration_manager*)this)->configuration_manager::m_cfgfiles, which is of non-class type int make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "class" from rconfig.h[269] declaration of m_cfgfiles, Added "class" to rconfig.h[225] typedef of cfgfiles_type. - The following error message was produced during compilation: tortoise% make make all-am make[1]: Entering directory `/home/peek/prog/rvm/src' g++ -DHAVE_CONFIG_H -I. -g -Wall -MT rconfig.o -MD -MP -MF .deps/rconfig.Tpo -c -o rconfig.o rconfig.cc rconfig.h:369: error: extra qualification job_parser:: on member read_job make[1]: *** [rconfig.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "job_parser::" from rconfig.h[369]. - The following error message was produced during compilation: g++ -DHAVE_CONFIG_H -I. -g -Wall -MT vaulter.o -MD -MP -MF .deps/vaulter.Tpo -c -o vaulter.o vaulter.cc table.h:124: error: extra qualification table:: on member col_width table.h:125: error: extra qualification table:: on member row_width table.h:126: error: extra qualification table:: on member col_height table.h:127: error: extra qualification table:: on member row_height make[1]: *** [vaulter.o] Error 1 make[1]: Leaving directory `/home/peek/prog/rvm/src' make: *** [all] Error 2 Removed "table::" from these lines. * Experimental: See if I can figure out why vault-locking fails. -- Michael S. Peek <peek@tortoise.tiem.utk.edu> Thu, 7 Jun 2007 11:09:10 -0400 rvm (1.01) stable; urgency=low * Added code to check to see that the vault is writable before running rsync. (In the event that something goes wrong with the filesystem, the mount may be changed to read-only. * Added a new global option: error-logging-level. When archiving a path for a job, the logging-level is used unless rsync returns an error, then the error-logging-level will be used for the subsequent retries of that path. * Fixed bug that would not resolve wildcard vault paths when only one path was found that matched the wildcard. * Fixed bug in vault_manager::get_archive_list() that would incorrectly construct the list of valid archive subdirectory names in a vault. * Added new global option: vault-locking. This option takes a boolean value. If enabled, rvm will lock the vault that it's using. Subsequent instances of rvm will (if run with vault-locking enabled) check to see if the vault is locked when selecting a vault. The behavior of rvm under vault-locking is as follows: - If a locked vault contains an archive with the same timestamp as the one being used by rvm then rvm exists with an error. - Otherwise, if a vault is locked, then that vault is ignored during the selection process and the next best vault is chosen instead. - If all vaults are locked then no vault is selected and rvm exits with an error. Rvm implements vault locking by creating a .rvm_lock file in the vault directory containing rvm's PID. When rvm checks to see if the vault is locked, it first checks to see if a .rvm_lock file exists, and if so, it then checks to see if a process is running with the PID listed in the .rvm_lock file. NOTE: If vault-locking is disabled then any existing locks created by other instances of rvm are ignored. (Be forewarned...) * Fixed bug that would allow jobs that generate an empty job ID string for status reports and the report log. If the generated job ID string is empty, then the job's jobname is used. A check is now performed after configuration that tests the job's generate_job_id() function to see if the string generated is empty. If it is, an error message is produced instructing the user to assign a descriptive jobname to the job in question. * Updated man page rvm.1 with any changes listed above that affect configuration and/or behavior. * Ran a spell checker on the man page -- Gads! I'm a terrible spellar. * Added an introduction to the log file stating program name and version. This is done by log_manager::init(). * Added new global commands: delete-old-log-files and delete-old-report-files. Both take a boolean value. If enabled, then when rvm deleted old archives from a vault, it will also search for and delete the log file and/or report file from the log-dir directory. -- Michael Peek <peek@tiem.utk.edu> Wed, 21 Jun 2006 11:49:19 -0400 rvm (1.0) stable; urgency=low * Initial Release. * Development continuing on a debian system as the old solaris systems are being depreciated. As such, for convenience sake, the changelog format will switch to the format used by debian package management tools. * When deleting a file or directory, check that the file/directory exists. If it does not, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if the file/directory still exists. It's possible that some user or some other running process just deleted it. If it does not exist, return without throwing an error. * When deleting a file or directory, and deletion failed, check to see if we have write privledges. This allows for a more descriptive error message. * A debian/ directory with files to generate a single, debian-native rvm package. * Code to Makefile.am to generate a debian package. * Code to Makefile.am to generate an RPM package. (Experimental) * Code to Makefile.am to generate a binary tgz package. * Contents of old changelog: 2005-04-15: Beta 0.90.4 Added: - RVM will now log a warning message when deleting old, incomplete archives. - vault_manager::get_archive_list() now only includes directories of archives with valid timestamp names, including incomplete archives. - error::push_back(const error& a_e), which will copy all the error instances from a_e to another error object. Fixed: - A bug in vault_manager::select() that prevented it from selecting the correct vault in the event that an archive of the given timestamps already existed in an incomplete form. - A bug in vault_manager::delete_oldest_archive() that prevented it from recognizing an archive of the current timestamp in incomplete form when deleting old archives in the event of archive overflow. - Many bug fixes in global_parser::parse_vault that prevented wildcard vault pathnames from being resolved and processed correctly. Among other things, this fix also prevents RVM from exiting with an error in the event that one of the wildcard pathnames resolved points to a directory with incorrect permissions. - Changed: std::string("stuff...") to: static_cast<std::string>("stuff...") The former line causes an error on some compilers. 2004-05-10: Beta 0.90.3 Todo: - Vaulter and archiver are supposed to work together to insure that an archive by the same timestamp isn't created on two different vaults. Somehow this got screwed up, because I have found this to be the case on one occasion. Check the logs and possibly modify the code based on what's found to insure that this doesn't happen again. Added: Changed: - The archiver now creates the archive directory with the extension ".incomplete". Once RVM completes archiving all jobs, the archive directory is renamed, removing the ".incomplete" extension. a) This allows the archive administrator to see at a glance if an archive has been finished; or if RVM is no longer running, that an archive is incomplete and possibly corrupted. b) In the event that RVM is still running when another instance of RVM is started, the second instance of RVM will know not to use the still-incomplete previous archive as a hardlink source for the new archive. c) Incomplete archives are not listed in the link catalog. - Cleaned up a lot of code associated with vault overflow handling. - Rvm no longer exits when a vault cannot be found. An error is written to stderr (since this is done in the configuration phase and the logger has not been initialized yet) and the vault path is ignored. This has prompted me to start designing modifications to the reporter and how it interacts with the other objects... (See Ideas's above) Fixed: - Added missing subdirectory assignment member functions and operators. This should fix a bug in vault_manager::delete_oldest_archive(). - Fixed a nasty bug in global_parser::parse_vault() that would not accept a full pathname to a vault(!?) (Don't know how that one slipped through the tests.) - Fixed numerous bugs in vault overflow detection and handling. - Added "\n" to logs generated by vaulter::delete_oldest_archive() - vault_manager::prepare() would delete more than one vault per execution when vault-overflow-behavior was set to delete-oldest. - Vault overflow would mysteriously disappear by the time rsync exited. Culprit was traced to the overflow event being caused by a temporary file created by rsync and then removed before exiting with an error condition. Once rsync exited the overflow situation would no longer exist and rvm would not attempt to delete an old archive. This would result in rsync being re-run and the same situation happening over again. Now if an overflow situation is detected a flag is set, and even if the overflow situation no longer exists after rsync exits rvm will still know to attempt to clean up the vault before retrying rsync. - If rsync exits with an error, and a vault overflow event has been detected, then the child process handling that job will not waste time attempting to retry that path, but will move on to the next path. - Fixed a bug in vault_manager::prepare() that would continue to delete vaults after a single vault had already been deleted. - If an overflow is detected, and vault-overflow-behavior is set to delete-oldest, and the oldest archive found at the time of execution has already been deleted, then attempt to continue archiving anyway. Do not delete any more old archives, and do not abort the current archive. Instead, let the jobs run and let rsync fail on it's own if there is not enough space. - Fixed a bug in vault_manager::prepare() that would not set the exit status to 2 (vault full) when the vault fills up and no more archive deletions can take place when vault-overflow-behavior is set to delete-oldest. - Fixed a bug where jobs exiting with an error are not rescheduled in the event that the vault is also found to have exceeded it's overflow threshold. - Fixed a bug where jobs exiting with an error, and the vault has not exceeded it's overflow threshold, and the job continues to sit in the queue. - Fixed a bug in job::generate_archive_path() that would create duplicate directory structures for paths that are copied (as opposed to paths whose *contents* are copied). This bug would be seen when a path was specified without a '/' character at the end. For instance: ... <job> ... path /home/peek/download/ddd-3.3.1.tar.gz ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ddd-3.3.1.tar.gz/ being created, and the file ddd-3.1.1.tar.gz being placed inside. This bug would also affect directories: ... <job> ... path /home/peek/download ... </job> ... This would result in the archive path name <archive-dir>/home/peek/download/ being created, and the directory download/ being placed inside. - Fixed a nasty bug in estring that would yield incorrect values for some signed integral types. - Fixed a minor bug in report generation. - Fixed a minor bug in error logging. - Fixed a bug that would allow the timestamp-resolution setting in a configuration file override any timestamp resolution setting derived from a --timestamp command line option. - Updated man page to reflect behavioral changes and lessons learned. :) 2004-05-08: Beta 0.90.2 Changed: - Miscilaneous spelleeng korrections Bugs Fixed: - Fixed a bug in rsync command line generation that the last bug fix introduced. Deja-vu. Also wrote a test to make sure it's generating what I think it is. (Should have done that the first time.) 2004-04-28: Beta 0.90.1 Changes: - Man page corrections as provided by Tony Sequeira. - Rsync uses --exclude-from and --include-from command line options, not --exclude-file and --include-file. Updated RVM man page to avoid confusion. Bugs fixed: - Fixed a bug in rsync command line generation -- again. I thought this one was licked some time ago, but I must have goofed somewhere. - estring::T_fraction_to_strings() would incorrectly convert some fractional values without paying attention to the set precision value. 2004-04-22: Beta 0.90 This release incorporates all features originally planned for RVM by the author, and concludes RVM's alpha development status. An extensive beta testing phase will now begin where RVM will be used daily in a production environment. New features may be added as requested by the OpenSource community and as time for development allows. As always patches for both feature requests and bug reports are welcomed by the author with open arms. - Added feature request: The rsync-options, when used in a job context, now adds to any rsync options specified in the last default context rather than overwriting any previous value. To override an rsync-options command specified in a default context the clear command must now be used. - Replaced safe_add and safe_mul functions with (what I hope is) the more robust, complete, generic, and faster safe_num class. - New table object, used by the report manager in formatting text reports. - Report manager completely rewritten. - Statistics reported by rsync removed from reports. (Output from rsync for runs involving hardlinks were erroneous, and thus RVM's reports would confuse and annoy RVM's simpleton author.) The author considered porting the analyzer from 0.79.1-alpha, but decided that it would only add to RVM's runtime. On the author's system, it currently takes over 18 hours to archive 334GB using two instances of rsync running in parallel on a virtual 200base-T connection consisting of two bonded 100base-T physical connections. The author feels that this is taking too long already, and is currently seeking ways to cut RVM's runtime. Maybe if enough people want it it will incorporated in a future release. - Added a new command: io-poll-interval. This command appears in a global context and accepts any non-negative integer value. This command instructs RVM to sleep some number of seconds between polling for I/O from children. - Modified job_archiver::mf_process_rsync_io() for more efficiency. - Forgot to modify the log_manager -- if a previous log file already exists by the same name, RVM will start appending numbers to the name until it finds a filename that doesn't already exist. The same modification has been made to the report_manager too. - Modified the log file names generated. Before, RVM would use: log.<timestamp> and report.<timestamp>. This has been reversed: <timestamp>.log and <timestamp>.report. The author liked having both reports together in an alphabetized directory listing. (The author is wondering how many people this will confuse.) Bugs fixed: - A patch submitted by Ezra Peisach fixes a bug that prevented RVM from constructing a valid command line when connecting to a remote rsync server without specifying the port. Platforms tested: Platform Config Compile Passes Tests ---------------------------------------- ------ ------- ----------------- i386-unknown-freebsd4.10 FreeBSD-4.8 OK OK OK i386-unknown-netbsdelf1.6.1 NetBSD-1.6.1 OK OK OK i686-pc-linux-gnu Debian-2.2 OK OK 1 test fails [2] i686-pc-linux-gnu RedHat-9.0 OK OK OK powerpc-apple-darwin6.8 OS-X-10.2 OK OK 1 test fails [1] sparc-sun-solaris2.9 Solaris-9 OK OK OK x86_64-unknown-linux-gnu SuSE-8-ES OK OK 1 test fails [2] [1]: test-rconfig-004 fails on Mac-OS-X-10.2 This test uses a set of configuration files that intentionally loop, causing RVM to re-read the same files over and over again util memory is exhausted. On this platform, when memory is exhausted, test-rconfig-004 fails at (backtrace): 1) _lookup_all_secure() 2) lu_getpwname() 3) getpw_internal() 4) getpw() 5) filestatus::path(std::string) This failure is not seen in OS-X-10.1. To the best of the author's knowledge this is an emergent feature of 10.2 that is caused by either a lack of memory or a lack of stack space. No error is thrown by test-rconfig-004, it merely exits with a segmentation fault after receiving signal EXC_BAD_ACCESS (could not access memory error). 10.2 seems to ignore the try {...} block that wraps the code that causes this mess, meaning that if such a lack of resource occurs in real life RVM will be unable to recover from the error. To the best of the author's knowledge, so long as a user does not run out of memory or stack space then RVM should function as desired. The author seeks a solution. [2]: test-rvm fails This has been seen on SuSE-8-ES running on an AMD64 Opteron, and on Debian-2.2 running on an i686. Test-rvm fails an assertion at line 580: in.is_open(). Errno is not set when the failure occurs. At this time the cause of failure is unknown. It is unknown whether or not RVM will function as desired on this platform. This behavior is not seen on Redhat-9.0 running on the same architecture as Debian-2.2. The author seeks a solution. To-do: - Comb through the code and make sure that all exceptions are being handled gracefully. (*sigh*) 2004-03-23: Alpha 0.80.2 This is a bugfix release. Bugs fixed: - archive_manager now throws an exception if used uninitialized. - Fixed an implicit conversion bug that would bite on some systems when allocating a new job_archiver using a configuration_manager::job_type::const_iterator as an argument. - catalog_manager now throws an exception if used uninitialized. (Fixed one point of usage where it was being used uninitialized.) - Errno behaves in different ways on different systems. Therefore the error object now takes an errno number as a parameter to the constructor rather than taking it directly from the errno variable. This allows greater control as errno can be used for errors that are expected to set the errno variable, and 0 can be used for errors that are not expected to set the errno variable. This provides a much more uniform behavior of the error object from one system to another. - error::str() now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - estring now defines the alphabet length used in base conversions to avoid possible data corruption. - estring now uses a char[] buffer and snprintf() to avoid a possible buffer overflow segfault. - log_manager now throws an exception if used uninitialized. - configuration_manager::clear() now resets the default log-dir to the value determined by the configure script rather than to the hard-coded value of "/var/log/rvm". - configuration_manager now throws an exception if used uninitialized. - rconfig.cc's parse_dirname() now checks for memory allocation error. - report_manager now throws an exception if used uninitialized. - Some tests altered to more accurately simulate an actual rvm run, code simplification of the *_manager classes ensued. - test-logger.cc expanded. - Some minor bug fixes to several tests that would prevent them from passing on various systems. - timer.cc's timer::make_string_() now checks for memory allocation error. - Modified timer::eta() to check for math overflow error. Modified one version of timer::eta() function to check that the timer has been both started and stopped before attempting calculation. - tstamp.cc's timestamp::str() now checks for memory allocation error. - vault_manager now throws an error if used uninitialized. - vault_manager::usage() now checks for math errors and writes an error to the log file is one is detected. (No longer causes rvm to die with an error.) - Fixed a bug in vault_manager::select() that could affect the max-free vault selection behavior. - vault_manager::select() now checks for memory allocation error. - Fixed a bug in fs.cc's mk_relative_path() that generated broken relative symbolic links. 2004-02-16: Alpha 0.80 Completely rewrote much of the code, too many changes to list them all. (In fact, with so many changes, 0.80 may have more bugs than 0.79.1...) Rvm should now compile and run under gcc 2.95.x. Configuration has been completely redesigned. Old configuration was based on a Qmail style schema using "control files". The new configuration method resembles something more akin to apache's httpd.conf. Many options can be specified on a job-by-job basis with defaults that can be set to be assumed by subsequent jobs. Rvm can now use rsync to archive locally-mounted files or files on an rsync server, as well as remote hosts accessed via RSH or SSH. Added a timeout option, after which, if no I/O is received by rvm from rsync, rvm will terminate the rsync process (and optionally re-try it). Rvm will now give different exit codes depending on differing error conditions that arise. Previous versions of rvm included an analyzer run by the child process to count the number and size of new files versus files hardlinked from an older archive. This analyzer is no longer present in rvm. Instead, rvm reads the output from rsync to obtain new vs. hardlinked files. However, the statistics gathered through this new method could prove to be confusing. If time permits, and the OpenSource community desires, then I may recode the analyzer back in for a future version. The report manager needs to be reworked. The error manager needs to be reworked. A lot of code needs minor work and general cleaning up. Much of the C++ code in rvm is probably in violation of the philosophical spirit of OO design. At the moment there is only one log file. In 0.79.1 there was one master log file and one log file for each client. However, in 0.80 RVM's design was moved from a client-based paradyme to the more abstract job-based paradyme. At this time it is not obvious to the author how to map each job to it's own log filename in an intuitive fashion, so the multiple log file scheme has been discontinued. The new log file scheme includes logging the PID and a uniq ID for each job on every line of the log file, this way output from a particular child process or a particular job can at least be grepped for. (If the OpenSource community cares enough about it, the author could add a log-dir command to the job context language.) Fixed a bug in the estring object. It could have been caught by a regression test, but wasn't because the tests are incomplete. Many, many more tests should be written, but the author simply has not had the time. Porting: The author has tried to write rvm to be as portable as possible. However, since the author's time and access to various systems is limited, there could by problems. At the time of this writing the test suite has been run on the following systems: a) PowerPC / MacOS-X 10.1 : All 13 tests pass b) PowerPC / MacOS-X 10.2 : 1 of 13 tests fails test-rconfig segfaults at: #0 0x9008ca74 in _lookup_all_secure () #1 0x90031e64 in lu_getpwnam () #2 0x9008b3f8 in getpw_internal () #3 0x9008b268 in getpw () #4 0x0005a7dc in filestatus::path(std::string) (this=0xbff84530, a_path={static npos = 4294967295, _M_dataplus = {<allocator<char>> = {<No data fields>}, _M_p = 0x3285cc "."}, static _S_empty_rep_storage = {0, 0, 357, 0}}) at fs.cc:726 The failure is in getpwuid() in the filestatus::path() function. The uid number passed to getpwuid has been verified to be correct, and a subsequent test of getpwuid from a separate program succeeds, which leads me to suspect a memory leak somewhere. (*gag*) c) Sparc / Solaris 8 : All 13 tests pass d) Sparc / Solaris 9 : 1 of 13 tests failed test-rvm throws the following error: *** INTERNAL ERROR Multiplication overflow error: 320646177970143232 * 100 in void safe_mul(T&, T) [with T = long long unsigned int] at rmath.h[56] Assertion failed: !thrown, file test-rvm.cc, line 263 Most likely this means that I'm not initializing a variable somewhere. e) AMD64 / SuSE 8 : All 13 tests passed f) i686 / Debian 2.2 : All 13 tests passed g) i686 / FreeBSD 4.8 : All 13 tests passed h) i686 / RedHat 9.0 : All 13 tests passed To-do: 1) Develop a tabular object that uses estring objects to lay out reports. There's a minor, asthetic bug in the current layout code, and I think this would be very helpful. 2) Rewrite the analyzer from 0.79.1 to gather statistics. Maybe the use of rsync's output versus statistics gathered by the analyzer should be a configurable option. 3) From (1) and (2), rewrite the reporter classes. 4) Clean up code. 5) Test, test, test, test, test... 2003-12-04: Alpha 0.79.1 Fixed a bug in timer object that would attempt to divide by zero if the elapsed duration was zero seconds. (Duh!) Modified and added to autoconf scripts to identify which flavor of statfs a system has, and what include files are needed to use it. Added tests to autoconf to determine whether a system uses struct statfs.f_namemax or struct statfs.f_namelen. Added an option to the timer object to force it to use Greenwich mean time instead of localtime. ...Silly test programs failing because people were in a different time zone. Modified test-timer.cc and test-report.cc to suit. (Duh!) 2003-11-24: Alpha 0.79 Moved configuration file retry-count to rsync-retry-count. Added the configuration file rsync-behavior to allow rvm to map different exit error codes from rsync to commands telling rvm how to handle them. For the errors that are retryable, rvm will retry up to rsync-retry-count times. Included tests in autoconf scripts to check for _FILE_OFFSET_BITS and _LARGE_FILES for Large File support. Lots of code cleanup, more still needed. Rvm is far from **optimized**, but it seems to work. Overflow code and many error-recovery routines still untested. 2003-10-31: Alpha 0.78 More work on backup.cc Revamped logger. Didn't wind up using it the way I had anticipated. Wrote help.cc which has version() to print a useless version message, and help() to print some useless help text. Updated the man page, spellchecked the man page. Rvm has not been rigorously tested yet. Feedback and patches are welcome. 2003-10-23: Alpha 0.77 Added --timestamp <timestamp> command line option to allow retry of a failed or interrupted previous archive attempt. Moved backup code to backup.cc Moved catalog code to catalog.cc Revamped backup code, squashed bug that prevented proper flushing of IO from child to parent. Still needs lots of work. 2003-10-17: Alpha 0.76 rvm.cc revamped for parallel processing of clients using the class execute. Report generation removed -- must be reworked for parallel code. Wrote a man page. Made a lot of changes to the code and to the autoconf scripts for various fixes. 2003-10-09: Alpha 0.75 Changed name to rvm, for Rsync Vault Manager. Vbk didn't really stand for anything, but I hadn't thought of anything better to name it at the time. Introduced the --relink command line option to have rvm regenerate the link catalog. Introduced class report for report generation, integrated into rvm.cc. Modified timestamp class to include a resolution variable, integrated into config.cc and rvm.cc. Completely revamped the way class config reads configuration files. Files can now contain comments, and line lengths are no longer limited. Modified the syntax of <logdir>/clients. Clients and pathnames are now specified as multiple lines of <client>:<path>, and there can be multiple lines with the same client name. This also resulted in a change in the way the config class stored this information. Squashed many bugs. 2003-09-30: Alpha 0.50 (Alpha test code works!) Code cleanup. Rewrite of class error. Class error now contains an internal flag noting whether or not the error is internal or external, and contains a stack trace of the error all the way back up to main(). Also introduced some macros for ERROR(), INTERNAL_ERROR(), ERROR_INSTANCE(), and TRY() -- which is a generic try{} block to push information onto the error's stack trace. Rewrite of all functions to utilize the new error object and macros. Introduced class logger. Logger is used to give a generic interface to logging services in such a way that log messages could be sent to any output device, whether it be std::cout, std::cerr, a file, a file descriptor, or another ostream object. Introduced mk_relative_symlink() in fs.cc, which will take two absolute paths, a source and a destination, and create the destination as a relative symbolic link to the source. Introduced the linkdir configuration file, which holds a path to a directory in which to generate a catalog of backups, by both timestamp and client, as relative symbolic links to their real locations in the vaults. Archiving code completely rewritten. Introduced the exclude and exclude-<client> configuration files. If either the exclude file or the exclude-<client> file exists, their complete pathnames are passed on to rsync using the --exclude-from=<file> command line argument. Vault preparation moved from the vault class in vault.cc to rvm.cc, and rewritten to utilize the logger. (If a previous archive is deleted to make room for a new archive, I think that action should be logged.) Fixed a bug in class timestamp that allowed a constant timestamp to change due to a misunderstanding of how localtime() worked. (The tm structure is now held as a static object in the class instead of as a link.) Class execute modified to allow the user more control over stdio redirection in the child. Introduced the to_string() template function to convert objects to a string, mostly used generating verbose error messages. Removed all the functions in fs.cc that used const char* arguments. (New to C++, I didn't realize that char* would be automatically converted to std::string...) 2003-08-29: Alpha 0.00 Document designs, concepts, and implementation strategy, test prototype code necessary to complete details of the design. -- Michael Peek <peek@tiem.utk.edu> Wed, 23 Nov 2005 14:53:05 -0500 # vim:ts=3:shiftwidth=3:et:

posted 136 days ago

Release Date:

Topics:

License:

Ratings and Reviews

Project Feed

rvm 1.05 file released: rvm_1.05.tar.gz

rvm 1.05 file released: rvm-1.05-README

rvm 1.05 file released: rvm_1.05_i386.deb

rvm 1.05 file released: rvm_1.05.dsc

rvm 1.05 file released: rvm-1.05-Changelog

rvm 1.05 file released: rvm-1.05-bin-i386.tgz

rvm 1.05 file released: rvm-1.05-bin-amd64.tgz

rvm 1.05 file released: rvm_1.05_amd64.deb

rvm 1.05 file released: rvm-1.05-2.x86_64.rpm

rvm 1.05 file released: rvm-1.05-2.i386.rpm

Rate and Review

Would you recommend this project?

Related Projects