Robinhood Policy Engine Files

Robinhood - Policy Engine and monitoring tool for large file systems.
====================================================================

RPMs available in the download section:

Common RPM (noarch), needed for all robinhood modes:
	rpms/robinhood-adm-2.4.0-2.noarch.x86_64.rpm

[TMP FS] Monitoring and space cleaning for scratch filesystem

Lustre 1.8 / RHEL5:	rpms/lustre1.8/robinhood-tmpfs-2.4.0-1.lustre1.8.el5.x86_64.rpm
Lustre 2.0 / RHEL5: 	rpms/lustre2.0/robinhood-tmpfs-2.4.0-1.lustre2.0.el5.x86_64.rpm
Lustre 2.1 / RHEL6: 	rpms/lustre2.1/robinhood-tmpfs-2.4.0-1.lustre2.1.el6.x86_64.rpm
Lustre 2.2 / RHEL6: 	rpms/lustre2.2/robinhood-tmpfs-2.4.0-1.lustre2.2.el6.x86_64.rpm
Lustre 2.3 / RHEL6: 	rpms/lustre2.3/robinhood-tmpfs-2.4.0-1.lustre2.3.el6.x86_64.rpm
Other POSIX FS / RHEL5: rpms/posix_fs/robinhood-tmpfs-2.4.0-1.el5.x86_64.rpm
Other POSIX FS / RHEL6: rpms/posix_fs/robinhood-tmpfs-2.4.0-1.el6.x86_64.rpm
Other POSIX FS / Fedora15: rpms/posix_fs/robinhood-tmpfs-2.4.0-1.fc15.x86_64.rpm

For other configurations, build from source tarball: robinhood-2.4.0.tar.gz

[BACKUP] Monitoring and file archiving

Lustre 2.0 / RHEL5: rpms/lustre2.0/robinhood-backup-2.4.0-1.lustre2.0.el5.x86_64.rpm
Lustre 2.1 / RHEL6: rpms/lustre2.1/robinhood-backup-2.4.0-1.lustre2.1.el6.x86_64.rpm
Lustre 2.2 / RHEL6: rpms/lustre2.2/robinhood-backup-2.4.0-1.lustre2.2.el6.x86_64.rpm
Lustre 2.3 / RHEL6: rpms/lustre2.3/robinhood-backup-2.4.0-1.lustre2.3.el6.x86_64.rpm

For other configurations, build from source tarball: robinhood-2.4.0.tar.gz

[WEB UI] Stats interface

webui/robinhood-webgui-2.4.0.tar.gz
webui/robinhood-webgui-2.4.0-1.noarch.x86_64.rpm

[Documentation]

doc/robinhood_v240_tmpfs-admin_guide.pdf
doc/robinhood_v240_tmpfs-tutorial.pdf

========== Change Log ========== 

Changes between version 2.3.4 and 2.4.0:
- [feature] rbh-du and rbh-find: "du" and "find" clones querying robinhood's database
- [report] new directory stats, top directories per dirent count, per avg file size
- [report] file size profiling: global, per user, per group, per fileclass...
- [report] sorting user/groups by size range (eg. percentage of files < 1G)
- [webUI] "size" section in webUI to display file size profiles
- [feature] Partial scans to update only a subset of the filesystem (also allows distributed scans)
- [packaging] rpm name 'robinhood-tmp_fs_mgr' changed to 'robinhood-tmpfs'
- [packaging] 'rbh-config' command moved to new RPM 'robinhood-adm'
- [report] refurbished rbh-report output format
- [policies] new criteria on file creation time
- [database] use innodb by default for MySQL engine
- [system] ability to detect "fake mtime" (mtime != actual modification time)
- [system] improved filesystem detection,
           using fsname or devid as FS identifier (config driven)
- [scan] can trigger external completion command when a scan ends
- [misc.] can use short config name instead of full path
          (eg. "-f <name>" instead of "-f /full/path/to/name.conf")
- [backup] directory and symlink recovery
- [lustre] port to Lustre 2.2 and 2.3
- [lustre] support for new Changelog record struct (lu-1331)
- [fix] max_rm_count=0 resulted in no rm (instead of unlimited)
- [fix] segfault in realpath() on Ubuntu
- [fix] unsigned arythmetic issue with MySQL 5.5

Changes between version 2.3.3 and 2.3.4:
- Faster and safer shutdown on SIGINT/SIGTERM
- Can use short config name instead of full config file path
  E.g. "-f myconf" instead of "-f /etc/robinhood.d/tmp_fs/myconf.cfg"
- Consider all non-dirs for classinfo (instead of files only)
- Implemented max_rm_count in hsm remove policy
- clearer messages about DB connection and retries
- added lu543 configure option (must be enabled if this patch
  is integrated to your Lustre distribution)
- Better block counting for purges
- backup/shook modes:
    - import of existing files from backend
    - entry state set to 'archive_running' during migration
    - recovery for entries with 'release pending' or 'restore running' state on startup
      (new parameter: check_purge_status_on_startup)
    - enable DB rebuild if it is lost
    - fix: symlink recovery
    - improvements of rbhext_tool_clnt/svr (timeout, traces, ...)
    - user.shook_state xattr changed to security.shook_state
      (to avoid users to change it)
- Fix: Don't consider 'released' entries for quota-like purge triggers
- Fix: migrate-group did migrate user
- Generate up-to-date template automatically at RPM installation
- 72 new regression tests (all policies conditions and config file parameters are tested)

Changes between version 2.3.2 and 2.3.3:
- [webgui] added FS name to page title and page header
- [webgui] fix: added missing file in RPM (.htaccess)
- [reports] new options for top-users/top-groups: --by-avgsize, --count-min, --reverse
- [reports] Lustre changelog stats in 'rbh-report -a'
- [policies] fix: 'tree' condition must match root entry
- [policies] fix: migration class matching at scan time
- [config] simpler scan parameter 'scan_interval'
- [config] fix: don't reload config of disabled modules on SIGHUP
- [config] fix: on SIGHUP, don't reload parameters specified on cmd line
- [database] retry on connection failure
- [stats] dump process stats on SIGUSR1
- [backup] clean special chars in archive names
- [backup] fix issues in symlink archiving
- [lustre] specific compilation option for jira's LU-543
- [misc] code cleaning, sanity checks, improved traces...

Changes between version 2.3.1 and 2.3.2:
- [webgui] Web interface (beta)
- [quota/alerts] Implemented quota alerts on inode count (users and groups)
- [reporting] New option --by-count for --top-users, to sort users by entry count
- [database] Support of InnoDB MySQL engine
- [database] MySQL 4 compatibility fix
- [bugfix](minor) handling DB deadlock error
- [bugfix](tweak) added acct parameters to default and template outputs
- [testing] big tests with 1M entries
- [backup] about backup mode (beta):
    - [bugfix](major) fixed error determining symlink status
    - [bugfix](minor) don't consider 'new' entries in deferred removal
    - [trace] display warning if mtime in FS < mtime in backend

Changes between version 2.3.0 and 2.3.1:
- [bugfix](major) Wrong accounting values if file owner changes
- [bugfix](major) SQL error for widely striped files
- [compat] Compatibility fix for MySQL servers between 5.0.0 and 5.0.32

Changes between version 2.2.3 and 2.3.0:
- [optim.] instant accounting reports (user/group usage, fs content summary, ...)
- [reporting] split user usage per group (--split-user-groups option)
- [reporting] split group usage per user (--split-user-groups option)
- [feature] new policy criteria for Lustre FileSystems: ost_index
- [reporting] detailed FS scan statistics in "rbh-report -a"
- [misc.] fast and clean abort on ctrl^c (during scan, migration and purge)
- [admin.] automatically disables features that are not defined in config file
- [admin.] "rbh-config backup_db" helper to create a robinhood DB backup
- [misc.] -V option displays Lustre version and release number
- [tweak] changed 'watermark' parameters to 'threshold'
- [tweak] changed 'notify_lw' and 'alert_hw' parameters to 'alert_low' and 'alert_high'
- [database] alternative port or socket file can be used for MySQL connection
- [database] limiting DB access rights for reporting command
- [bugfix](major) fixed inconsistent pool names
- [bugfix](minor) kill -HUP terminated the process if no trigger was defined
- [bugfix](minor) 'unknown' status not correctly filtered in '--dump-status' report
- [bugfix](tweak) added 'reload' in short help of SLES init script
- [misc.] code cleaning, error message cleaning, removed some obsolete code
- [feature] new robinhood flavor to track modifications in a Lustre v2 filesystem, and backup data to an external storage (current status: Alpha testing only).
  As part of this feature:
        - soft rm + command to retrieve removed files
        - disaster recovery command
        - "--migrate-file" option to archive a single file
        - pre-maintenance mode to smoothly backup the whole filesystem content before a due date.


Changes between version 2.2.2 and 2.2.3:

- [feature] periodic purge trigger
- [feature] options for controlling trigger notifications
- [doc] pdf documentation updated

Changes between version 2.2.1 and 2.2.2:

- [bugfix] (major) fixed "duplicate key" errors
- [bugfix] (major) FS scan sometimes blocks on Lustre 2
- [misc.] integration to automatic testing suite (Hudson)

Changes between version 2.2.0 and 2.2.1:

- [feature] new purge command: --purge-class to apply purge policy on files in a given class
- [feature] new migration command: --migrate-class to apply migration policy on files in a given class
- [feature] support of syslog for logging
- [report cmd] Added summary line to all reports, with total nbr entries and volume.
- [report cmd] Added '-q' option to hide headers and footers in reports.
- [optim.] changed primary key format to reduce DB requests
- [misc.] new command 'repair_db' in rbh-config, to fix tables after a MySQL server crash.
- [compat.] Support for Lustre MDT changelogs on Lustre v2.0 final
- [compat.] port to FreeBSD
- [admin] added 'reload' action to init.d script
- [misc.] a gap in OST index list should displays a warning, not an error
- [pkg] common spec file for both el4, el5 and el6
- [bugfix] handling large UNIX groups (>4k) and long lists of alt groups.
- [bugfix] retrieving Lustre pool fails with error "Unsupported Lustre magic number"
- [bugfix] wrong class matching on OST pools when scanning
- [bugfix] unescaped SQL strings caused error for filenames with single quotes
- [bugfix] error in init script when RBH_OPT contains several options

Changes between version 2.1.5 and 2.2.0:

- [feature] fileclass union/intersection/negation
- [feature] rbh-report displays last matched fileclass
- [feature] new reporting command '--class-info' generates fileclass summary
- [feature] new reporting option '--filter-class' to dump entries per fileclass
- [feature] alert batching: send a mail summary instead of 1 mail per matching entry
- [feature] alert improvements: named alerts, tweak changes
- [feature] special wildcard '**' in 'path ' or 'tree' conditions matches any count of directory levels
- [feature] quota-like purge triggers fully implemented (on group or user)
- [feature] triggers on used inode count in filesystem
- [feature] '--check-triggers' option to check triggers without purging files 
- [feature] notification can be sent when a high watermark is reached (for triggers)
- [feature] rbh-config helper now supports batch commands
- [feature] Lustre 2.0 ready
- [optim.] configurable fileclass periodic matching to reduce calls to filesystem
- [optim.] configurable attr/path periodic update in DB to reduce calls to filesystem
- [bugfix] explicit trace when readdir fails
- [bugfix] issue when filtering on fields with NULL values in DB
- [bugfix] check migration timeout on last effective action, not on last queued entry
- [bugfix] name-based conditions complaining about missing auto-generated fields
- [bugfix] race condition when appplying policy lead to handle the same entry several times
- [bugfix] removing removed directories from database for recursive rmdir policies
- [misc.] added documented file in /etc/sysconfig for robinhood service parameters
- [misc.] changing source directory layout
- [misc.] documentation update

Changes between version 2.1.4 and 2.1.5:

- Major bug fix: incomplete database content after scan

Changes between version 2.1.3 and 2.1.4:

- New recursive rmdir policy (for TMP_FS_MGR purpose)
- changed default value for max_pending_operations
  (unlimited value could result in excessive memory usage)
- removing useless fields and redundant information in database
- rh-* commands renamed to rbh-*, to avoid conflicts and confusions
  with RedHat commands.
- check conflicting flags in configure

Changes between version 2.1.2 and 2.1.3:

- SQLite support (should only be used for testing purpose or small filesystems)
- Support of relative paths in 'path' and 'tree' conditions
- Migration timeout mechanism
- Prompting for database admin password in rh-config script

Changes between version 2.1.1 and 2.1.2:

- New reporting commands: Dump all files (--dump-all) and dump files
  by status (--dump-status).
- New configuration helper script: "rh-config"
- Made RPM relocatable
- BUG FIX: wrong scan duration when using volume-based purge triggers
- Lustre-HSM: Checking previous migrations status when restarting
- Lustre-HSM: CL_TIME record support (bz 19505)
- Lustre-HSM: multi-archive support (archive_num)
- Lustre-HSM: new --sync option (immediately archive all modified files)
- Lustre-HSM: changed --handle-events action switch to --readlog
- Fixed SLES portability issues

Changes between version 2.1.0.beta2 and 2.1.1:

- Added new report options: --dump-ost, --dump-user, --dump-group
- Added --filter-path option to reporting tool.
- TMP FS MGR purpose ported to Lustre 2.0-alpha5 (including changelog
  support).
- documentation updates (in doc/admin_guides)
- Each purpose has its own service and binary names,
  to make it possible to install and run several robinhood with
  differents purposes on the same machine.
- Added '--disable-lustre' compilation switch for disabling Lustre specific
  features
- Added '--disable-fid-support' compilation switch, to force addressing
  entries by path, not by their Lustre fid.
- Integration of new purpose "SHERPA" (software suite for cache management)
- Generated RPM name includes lustre version it was built for.
- report command displays help if is started without option.

Changes between 2.1.0.beta1 and 2.1.0.beta2:

- Extended attributes support in policy definition
- 32 bits plateforms compatibility fixes
- Fixed bug when using mysql4
- Added parameter to force changelog polling
- Fixed minor compilation warning
- commands now search for config file in /etc/robinhood.d if no config file is
  given on command line

Changes from v2.0.1 to 2.1.0.beta1:

- added '--dry-run' option instead of "simulation_mode" parameter in config file
- added '--once' option, to perform an single pass of a given policy or action
  and exit (same as '--one-shot' option).
- Compatibility fixes for MySQL 4 and 5
- Fixed dependencies on lustre include files.
- Compatibility fixes for 32 bits platforms

Lustre-HSM specific features:
- Porting to the new changelog interface (handling changelog records as
  structures instead of text, and using CHANGELOG_FLAG_FOLLOW
  and CHANGELOG_FLAG_BLOCK options)
- Adapting to changes in changelog timestamp (secs+nano instead of jiffies)
- Use fid as primary key in database schema (for better performance)
- Added calls to llapi_hsm_request() to trigger migration, release, removal
  in HSM.
- Customizable migration hints to be passed to the copytool
- Command line options to trigger manual migrations (by user, by OST...)
- Deferred removal in HSM
- Taking HSM file status into account (dirty, released, ...)
- HSM event support
- Changelog flag support (for UNLINK and HSM event)
- Added '--ignore-policies' option to perform migration/purge to all eligible
  files without checking policy conditions.

Changes between v2.0-beta2 and v2.0.1:

- New policy definition semantics, using filesets
- Multiple fileset/policy associations
- Several changes in configuration syntax, to avoid confusions
- Support of OST pool names (on Lustre) for fileset definition and policies
- Optimizations of policy application
- Added features for Lustre-HSM