bfm Files

# BFM - Big File Management

## Overview

This tool is intended to manage big files outside of your VCS but in your work
area. It's like mercurial or git largefile extension but independend of any VCS.

Instead of the big file itself only a list of filenames with their
checksums (md5) are committed to the VCS. The BFM can check for	missing or
outdated bigfiles and downloads them from a server using CURL or CP.

The developer has not to care about e.g. using the correct version of an SDK
or big third party library since these things are downloaded in the correct
version which is tracked in the BigFile MD5.

Downloaded files can be located directly in the work area (same directory as the
bigfiles.txt checksum file) or at a specified output directory (option -O) or
in a cache directory to share the downloaded files between multiple work areas.


## Changes

version 0.15
- new option -O outputdir
- new option -x and --bfmcache
- support this option in check, show and download

version 0.14
- environment variables for defaults
- verbose debugging output
- small fixes
- add starter scripts

version 0.13
- upload.py respect file subdirectory
- bugfix to calchash function
- replace "grep --quiet" with "grep -q"

version 0.12
- upload via http if server-url match upload.py

version 0.11
- set filepermission to writeable on extraced archiv to have delete rights
- write info and debug output on stdout, only errors go to stderror

version 0.10
- add option --calchash to calculate md5 hash and --hash to specify a fixed hash for serverfilename
- support option "-" to read filenames, dirs from stdin
- support syntax @filename to read list of files/dirs from that file

version 0.9
- handle windows linefeeds in bigfiles.txt correctly
- add option to INCLUDE and EXCLUDE files by pattern
- add option to run in parallel multiple downloads

version 0.8
- handle spaces in file- and directory names correctly
- speed up by doing less subprocess calling of sed and more in bash string replacement
- add Makefile for bundling output
- add md5 of subdirectory browsing in debug output (helps while looking for diffs)

version 0.7
- add option -D and --dir to support directory handling
- check md5 after upload
- support subdirectories on server by option -P
- add argument of level to option -R
- fixed problem with no read permission on downloaded directory for md5 verification
- shorten names of local directory md5 cache filenames

version 0.6
- add option --show to list managed filenames (used by buildtools like cmake)
- add option -R to run recursivly looking for bigfiles.txt

version 0.5
- add option -r to remove files from bigfiles.txt
- ignore bigfiles.txt on upload even if specified on commandline (through pattern *)
- add option --changed to mark files as locally changed - permit updates
- exit with error code on failed downloads
- abort if server upload dir doesn't exists

version 0.4
- add junit like tests in test/runtest.sh
- add BFM_CACHEDIR variable and upload to cache after download

version 0.3
- add option -n to only show actions without execution
- add debug2 function for additional verbosity using multiple -v arguments
- use .config prefix on settings in bigfiles.txt
- add option --touch to touch files after download
- changed some information outputs

version 0.2
- fixed upload command with CURL

version 0.1
- first version of this tool
- working on OSX and Windows/Cygwin, but should on Linux too
- require bash, curl, md5/md5sum and some other basic commands (e.g. cp, rm, ...)