How to estimate time vs size?

Mark C
2004-11-28
2012-12-08
  • Mark C

    Mark C - 2004-11-28

    I am about to compress about 100Gb of data to allow for better archiving but most of all faster per file ftp download.

    I have experimented with some of the compression options and found some interesting results. (mainly some options increase comression time by 100-500% but compressed size only decreases by 1-10% over the fast option)

    What I want to do is compile a list of compressed size and time to compress for all the various options and types.

    Basically to find the best speed vs compressed size option for my type of data.

    Anyone got any idea how to do this?
    Anyone got a script I could use as a shell for this excercise?

    Cheers for any advice you can give.

    Mark

     
    • Mark C

      Mark C - 2004-12-02

      can i be rude and bump this up.

      Cheers,

      Mark

       
    • Puremagic

      Puremagic - 2004-12-02

      This is a script that might help in perl
      (yes i know it can be done more... nice
      but i was in a hurry)

      @method = ("hc4","hc3","pat4h","pat3h","pat2h","pat2","pat2r","bt4b","bt4","bt3","bt2");
      @dict = (1,2,3,4,6,8,12,16,24,32,48,64,96,128,192);
      @word = (8,12,16,24,32,48,64,96,128,192,255);
      $output = ">>output.txt";
      open (OUTPUT, $output);
      $time = localtime;
      print OUTPUT "$time\n";
      $name1 = "work\\temp.7z";
      $name2 = "output\\temp.7z";
      $filesize = "10000000000";
      for(@method)
      {
          $method = $_;
          $trigger = "";
          for(@dict)
          {
              $dict = $_;
              $dict = $dict . "M";
              for(@word)
              {
                  $word = $_;
                  system "7z a -t7z \"$name1\" \"input\\*\" -m0=LZMA:mf=$method:d=$dict:fb=$word:a=2 -ms=on -mf=on -mhcf=on -mhc=off\n";
                  print OUTPUT "7z a -t7z \"$name1\" \"input\\*\" -m0=LZMA:mf=$method:d=$dict:fb=$word:a=2 -ms=on -mf=on -mhcf=on -mhc=off\n";
                  $size1 = -s $name1;
                  if($size1 >= $filesize)
                  {
                      system "del \"$name1\"";
                  }
                  else
                  {
                      system "copy /Y \"$name1\" \"$name2\"";
                      print OUTPUT "OLD: $filesize\tNEW:$size1\n";
                      $filesize = $size1;
                      system "del \"$name1\"";
                  }
              }
          }
      }
      $time = localtime;
      print OUTPUT "$time\n";
      close OUTPUT;
      print "All done!\n";

       
    • Mark C

      Mark C - 2004-12-02

      Puremagic thats well.... pure magic lol

      Cheers,

      Mark

       
    • Puremagic

      Puremagic - 2004-12-02

      lol well forgot to write you need to change some things
      if used on linux/unix

      Hope you got some help from it anyway:)

      regards
      /PM\

       
    • Nobody/Anonymous

      you can add "lc" "lp" "pb" settings too

      MAAD

       
    • Puremagic

      Puremagic - 2004-12-04

      yeah im rewriting the script
      i post it when im done.
      i recommend using the script on small files of the same
      type as the large file to get best compression for
      the large file:)

      /PM\

       
    • Mark C

      Mark C - 2004-12-04

      had a go at running it on cygwin but it fails with some cryptic errors.

      not being a perl man any chance of simple run down of how it works.

      cheers!

       

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks