Metalink Verification

Help [OLD]
Joshua
2009-04-02
2012-09-28
  • Joshua
    Joshua
    2009-04-02

    I've been playing with the Metalink downloading and verification. From what I've observed looking at the logs Aria2 creates when downloading with Metalink if Chunk Checksums are present in the Metalink file Aria2 checks the Chunk Checksums as the file is downloaded, and then assumes that if all of the Chunk Checksums are correct, than the entire file must be correct. And if no Chunk Checksums are present than Aria2 uses the Sha1 hash to check the entire file.

    Are my observations correct? And if so is there a way to make Aria2 also use the Sha1 hash on the entire file to double check the file's integrity after downloading with Chunk Checksums? Or is that just not necessary? Thank you for your help.

     
    • tujikawa
      tujikawa
      2009-04-16

      I've not decided to apply patch in the next release.
      I think double hash checking is overkill for most users.
      If many users likes this patch, then I'll consider to make it into the main repository.

       
    • tujikawa
      tujikawa
      2009-04-02

      Your observation is correct.

      I think practically chunk checksum is good enough.
      The hash value of damaged piece of data is unlikely to match the (wrong) hash value in metalink file.

      If you want to double check, then instead of calculate hash value for entire file, run aria2 with -V option to re-check chunk checksum, because you have to repair download with -V option if entire hash check failed.

       
    • Joshua
      Joshua
      2009-04-02

      I'm really overly cautious when it comes to file integrity, but only because I've been burned by it so many times. The chunk checksum is probably fine, but there is that small chance that somehow either by software or hardware fault the file gets corrupted while being processed from chunks to the whole file. Or maybe I just don't understand how Aria2 works well enough to see why that's just not possible. Either way it would be great just for peace of mind if there way an option to checksum the entire file after downloading via metalink. Should I add that to the feature request list?

       
    • Joshua
      Joshua
      2009-04-03

      I'm really overly cautious when it comes to file integrity, but only because I've been burned by it so many times. The chunk checksum is probably fine, but there is that small chance that somehow either by software or hardware fault the file gets corrupted while being processed from chunks to the whole file. Or maybe I just don't understand how Aria2 works well enough to see why that's just not possible. Either way it would be great just for peace of mind if there way an option to checksum the entire file after downloading via metalink. Should I add that to the feature request list?

       
    • tujikawa
      tujikawa
      2009-04-04

      I know earlier version of aria2 had bugs that lead to corrupted downloads, but the recent release fixed them and I have no complaints about the corrupted download lately.
      In terms of file system crash, there are nothing for aria2 to do.
      Still there is a chance to file corrupted after calculating entire checksum because of bad sectors, etc. You need to check file hash regularly.

      Anyway, I think the changes to enable both chunk checksum and entire one is a few line of patch(needs more lines to add option to enable/disable this option). I can upload it when you open a request tracker.

       
    • Joshua
      Joshua
      2009-04-14

      Thank you for this patch. I don't think I'm up to applying the patch and then compiling the source, but I will try. Do you think this patch will make it into the next release?