Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#11 Making hard links if database not change

closed
nobody
None
5
2011-09-09
2007-11-09
Javier Barroso
No

Hi,

My patch add a new option to your script.

The option is HARDLINK

If it is true, a new directory ($BACKUPDIR/md5sum) will be populate with files like the follow example:

cat md5sum/mydatabase.md5sum
06c0417dc06a3fda1a93ecf70fc81ff4 /backups/weekly/DATABASE/DATABASE_week.41.2007-10-13.sql.gz

1ee297259f10a01ba155b321d50c8193 is "DATABASE_week.41.2007-10-13.sql" md5sum

A new function store do the hack.

Calls to compression compression functions are replaced by store function.

if HARDLINK is not set, store is like compression

Thanks for automysqlbackup

Discussion

  • John Sivak
    John Sivak
    2009-07-29

    Big issue with this patch if your using MySQL 5.0.27 or newer; the mysqldump command adds a comment to the end of the file that includes the date when the dump completed. This makes every dump different, even then the data has not changed.

    You'll need to add the "--skip-dump-date" to the $OPT for mysqldump to avoid this issue.

     
  • John Sivak
    John Sivak
    2009-07-29

    I forgot to mention: this issue is with the version of mysql client/mysqldump (not the server). I'm dumping databases from a MySQL 4.1 server using a 5.0.x MySQL client/mysqldump on a different box.

     
  • Javier Barroso
    Javier Barroso
    2009-07-30

    Don't count last line if it is "Dump completed on ..."

     
  • Javier Barroso
    Javier Barroso
    2009-07-30

    Hello jsivak, thanks for your feedback.

    I forgot update this patch (locally I fixed it some time ago)

    I attached a new version, with this issue.

    Not md5sum "file" but

    sed '\${/Dump/d}' | md5sum -

    Regards,

     
  • Hmm.. Wouldn't it be better to version check the mysqldump command and then use the "--skip-dump-date"? I guess I'm not trusting MySQL to keep the formatting and position of the "-- Dump complete.." line the same for future versions of mysqldump..

    And a different question: I don't see in the script/patch how the md5sum files are ever cleared/emptied. Do you do this manually periodically?

    Thanx.

     
  • John Sivak
    John Sivak
    2009-07-30

    (the last comment was from me.. I forgot to login.. grrr..)

     
  • Hi,

    The patch delete obsolete lines when it find a file who match but that doesn't exists:
    > while [ \( ! -e "$file_dup" \) -a -n "$file_dup" ]
    > do
    > sed -i "/ ${file_dup//\//\/}$/d" $MD5DIR/$DBNAME.md5sum
    > file_dup=$(awk "/^$md5 /"' {print $2; exit}' $MD5DIR/$DBNAME.md5sum)
    > done

    But yes, It is not complete. When you delete one file FILE, line from .md5sum file should be deleted too.

    With find it could be something like (until next week I don't have time to test and improved the next sentence) next example.

    For example deleting obsolete daily backups:
    find $BACKUPDIR ! -mtime -7 -wholename "*daily*" -printf "%h %f\n" -delete |
    while read dir file;
    do
    dbname=$(basename $(dirname $dir))
    sed -i '/'"$file"'$/d' $MD5DIR/$dbname.md5sum
    done

    Regards,

     
  • Javier Barroso
    Javier Barroso
    2009-08-01

    idem, the last comment was posted by me :)

     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    • status: open --> closed
     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    Well the functions for remove, add, parse such "manifest" files are not nearly complete. And md5sum is not really helpful, since there is datetime information in the sql file, from when the backup is. Therefore just use the new differential backup option, which uses diff to achieve just what you want. It will be released with the next beta version.

     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    Okay, I read the wrong file - the older one it seems - and the newer one seems to remove the datetime line. No guarantee however on the stability of this code. Perhaps we can discuss, what this is supposed to do:
    sed '\${/Dump/d}' "$1"
    If I do
    echo -e "test\nDump" | sed '\${/Dump/d}'
    I get the error sed: -e expression #1, char 11: unterminated address regex.
    Shouldn't this be
    sed '/^-- Dump/d' "$1" ?
    Or since it is the last line ...
    sed '$d'

     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    • status: closed --> pending
     
  • Javier Barroso
    Javier Barroso
    2011-09-09

    Sorry, I don't know how that was working in my old company, I cannot access to check there ...

    But

    sed -e '${/Dump/d}' | md5sum -

    do the trick (how hell I wrote \ before $, maybe I was playing with double quotes before ...

    $ == last line of the file
    {
    /Dump/d # remove line if /Dump/ is found
    }

    I don't know if the format has changed

    Thank you for maintain live automysqlbackup!

     
  • Javier Barroso
    Javier Barroso
    2011-09-09

    • status: pending --> open
     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    I'll just close it ... this feature in a way, will be available - as already mentioned - in the next beta version.
    Thank you for explaining that. Surely without the \ it makes sense ... didn't think of that.

     
  • PittaGurneyi
    PittaGurneyi
    2011-09-09

    • status: open --> closed