#42 aespipe compatibility

closed
nobody
None
5
2011-09-06
2011-08-20
dan
No

Hi,

I have a few questions about compatibility with aespipe. I am writing a C#.NET application which needs to generate an AES encrypted file (using AESCryptoServiceProvider). This file needs to be decrypted on a linux machine, and I'd like to use aespipe.

My main question is about the initialization vector (IV). Assuming I'll be using single-key mode and reading the key from a cleartext file, where does aespipe look for the IV? At the beginning of the encrypted file?

I would ultimately like to use multi-key-v3 for these files, but I honestly haven't looked too much into this. Would this even be possible? I'm assuming I would have to use AESCryptoServiceProvider to encrypt each block separately with a different key (up to 64) and then include am md5 hash of the IV (common to each?) as the 65th key.

I'm sorry if I come off as completely inexperienced with AES....I am :-)

I appreciate any help you can give me, thanks!

-Dan

Discussion

  • Jari Ruusu
    Jari Ruusu
    2011-08-21

    CBC-chains are 512 bytes long. Each 512-byte chain is
    independent of other chains as this allows random read/write
    access to any disk sector. IV is not stored in output/input
    file. IV is always computed from from file/storage offset
    and other info, and is a 128-bit number in little-endian
    byte order.

    In single-key mode, IV is a simple index to 512-byte sector
    number. 0 for first 512-byte chain, 1 for second 512-byte
    chain, 2 for third 512-byte chain, and so on. For more info
    see the compute_sector_iv() function in aespipe source.
    Single-key mode is weak/exploitable and should not be used.

    In multi-key-v2 and v3 modes, the IV is computed as MD5 sum
    of all data bytes of the 512 sector except first 16 bytes,
    and 56 least significant bits of sector number index. v3
    mode also prepends 128 bits of 65th key as input to MD5, so
    the MD5 computation starts with unknown initialization
    values. For more info see the compute_md5_iv_v3() function
    in aespipe source. v2 and v3 modes use 64 AES keys, first
    key for first 512-byte chain, second key for second 512-byte
    chain, and so on.

    Passphrase is hashed with SHA256, SHA384, or SHA512 to
    create one AES key. "cleartext file" means that file's
    contents are not wrapped in gpg encryption, but the
    passphrase string in a file is still hashed to create a key.

     
  • dan
    dan
    2011-08-22

     
    Attachments
  • dan
    dan
    2011-08-22

    Hi,

    Thanks for the help!

    That got me off to a pretty good start. Can you take a look at the code I've written? I think there may be a mistake in my algorithm...aespipe isn't decrypting files created with it. I'm using a cleartext file with 65 passphrases.

    Thanks!
    -Dan

     
  • dan
    dan
    2011-08-22

    One thing I noticed is that I was prepending the full SHA256-bit 65th key before calculating the md5sum for the IV, where you say to prepend 128bits...Are these the least significant bits?

     
  • dan
    dan
    2011-08-22

    Here's my modified code to calculate the IV:

    private static byte[] getSectorIV(byte[] sector, long sectorIndex)
    {
    byte[] key65 = new byte[16];
    Array.Copy(getKey65IV(), 16, key65, 0, key65.Length);

    byte[] sector496 = new byte[496];
    Array.Copy(sector, 16, sector496, 0, sector496.Length);

    byte[] sectorIndexBytes = new byte[7];
    Array.Copy(BitConverter.GetBytes(sectorIndex), 0, sectorIndexBytes, 0, sectorIndexBytes.Length);

    byte[] unhashedIV = new byte[key65.Length + sector496.Length + sectorIndexBytes.Length + 8];

    Buffer.BlockCopy(key65, 0, unhashedIV, 0, key65.Length);
    Buffer.BlockCopy(sector496, 0, unhashedIV, key65.Length, sector496.Length);
    Buffer.BlockCopy(sectorIndexBytes, 0, unhashedIV, key65.Length + sector496.Length, sectorIndexBytes.Length);

    UInt32 bits = 4024;

    Array.Copy(BitConverter.GetBytes(bits), 0, unhashedIV, unhashedIV.Length - 8, 4);

    for (int i = 0; i < unhashedIV.Length; i++)
    {
    Console.WriteLine(i + ": " + enc.GetString(unhashedIV, i, 1));
    }
    Console.WriteLine();

    MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();

    byte[] iv = md5.ComputeHash(unhashedIV);

    return iv;
    }

     
  • Jari Ruusu
    Jari Ruusu
    2011-08-22

    aespipe passphrase string hashes have additional twists. For
    single-key mode, hash output is truncated to AES key size.
    For multi-key-v2 mode, first byte of hash output is always
    XORed with 0x55, and hash output is truncated to AES key
    size. For multi-key-v3 mode, first byte of hash output is
    always XORed with 0xF4, and hash output is truncated to AES
    key size. These XORs are intentional incompatibility to
    prevent accidental partially correct decryption in wrong
    IV-computation mode.

    In multi-key-v2 mode, IV is straight MD5 of last 496 bytes
    of sector data and 7 bytes of sector number. As in:

    md5_initialize()
    md5_add_data(last 496 bytes of sector data)
    md5_add_data(7 bytes of sector number)
    md5_finalize()

    In multi-key-v3 mode, IV computation is sort of mutated
    version of MD5 that does not count the length of the 65th
    key bits at all. As in:

    md5_initialize()
    md5_add_data(first 16 bytes of 65th key)
    md5_add_data(48 bytes of zeroes)
    md5_save_128_bits_of_internal_state("partialMD5")

    Above needs to be computed only once. So, for each sector
    only this code needs to be repeated:

    md5_initialize()
    md5_restore_128_bits_of_internal_state("partialMD5")
    md5_add_data(last 496 bytes of sector data)
    md5_add_data(7 bytes of sector number)
    md5_finalize()

    So in both multi-key-v2 and multi-key-v3 cases MD5 input
    data length is 4024 bits (496 + 7 bytes). multi-key-v2 MD5
    IV computation starts with known "standard" internal state.
    multi-key-v3 MD5 IV computation starts with unknown internal
    state that depends on 65th key.

     
  • dan
    dan
    2011-08-22

    Thanks!

    I've made the changes you mentioned, but I'm still not having any luck getting it to decrypt so I must still be missing something :-(

    Here are my updated functions. Please let me know if you notice anything else!

    Thanks,
    Dan

    private static byte[] getSectorIV(byte[] sector, long sectorIndex)
    {
    byte[] key65 = new byte[64];
    Array.Copy(getKey65IV(), 0, key65, 0, 16);

    byte[] sector496 = new byte[496];
    Array.Copy(sector, 16, sector496, 0, sector496.Length);

    byte[] sectorIndexBytes = new byte[7];
    Array.Copy(BitConverter.GetBytes(sectorIndex), 0, sectorIndexBytes, 0, sectorIndexBytes.Length);

    byte[] unhashedIV = new byte[sector496.Length + sectorIndexBytes.Length];

    Buffer.BlockCopy(sector496, 0, unhashedIV, 0, sector496.Length);
    Buffer.BlockCopy(sectorIndexBytes, 0, unhashedIV, sector496.Length, sectorIndexBytes.Length);

    //UInt32 bits = 4024;

    //Array.Copy(BitConverter.GetBytes(bits), 0, unhashedIV, unhashedIV.Length - 8, 4);
    //unhashedIV[unhashedIV.Length - 8] = bits;

    ASCIIEncoding enc = new ASCIIEncoding();
    /*Console.WriteLine(enc.GetString(unhashedIV));
    Console.WriteLine("Length: " + sectorIndexBytes.Length);

    for (int i = 0; i < sectorIndexBytes.Length; i++)
    {
    Console.WriteLine(i + "/" + sectorIndex + ": " + sectorIndexBytes[i]);
    }*/

    for (int i = 0; i < unhashedIV.Length; i++)
    {
    Console.WriteLine(i + ": " + enc.GetString(unhashedIV, i, 1) + "/" + unhashedIV[i]);
    }
    Console.WriteLine();

    MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();

    md5.TransformBlock(key65, 0, key65.Length, key65, 0);
    md5.TransformFinalBlock(unhashedIV, 0, unhashedIV.Length);

    return md5.Hash;
    }

    private static byte[] getSectorKey(int index)
    {
    string keyfile;

    using (FileStream keyFS = new FileStream(keyPath, FileMode.Open, FileAccess.Read))
    {
    StreamReader keySR = new StreamReader(keyFS);

    keyfile = keySR.ReadToEnd();
    }

    string[] keyfileArray = keyfile.Split(new string[] {"\r\n"}, StringSplitOptions.RemoveEmptyEntries);

    byte[] passBytes = Encoding.ASCII.GetBytes(keyfileArray[index]);

    //if (index != 64) Console.WriteLine(keyfileArray[index]);
    //SHA256CryptoServiceProvider sha256 = new SHA256CryptoServiceProvider();
    SHA256Managed sha256 = new SHA256Managed();

    byte[] passHash = sha256.ComputeHash(passBytes);

    byte[] keyHash = new byte[16];
    Array.Copy(passHash, 0, keyHash, 0, keyHash.Length);

    keyHash[0] ^= 0xF4;

    return keyHash;
    }

     
  • Jari Ruusu
    Jari Ruusu
    2011-08-23

    I believe md5.TransformBlock / md5.TransformFinalBlock count
    the length of data normally. multi-key-v3 IV computation
    requires that for first 64 bytes of input to MD5 the length
    of data is "not counted".

    aespipe and loop-AES implementations speed optimize that
    first 64 byte MD5 input computation by computing that only
    once.

    I am still wondering why you are trying to encrypt data in
    your own code. How about creating a pipe to gpg program, and
    letting gpg handle the crypto.

    $ echo "password" >foo1
    $ echo "data" | gpg --symmetric --passphrase-file foo1 >foo2
    $ gpg --decrypt --passphrase-file foo1 <foo2 >foo3

     
  • dan
    dan
    2011-08-23

    So that basically means that when hashing the 496+7 bytes, the md5 initializer constants (usually 0x67452301 , 0xEFCDAB89, 0x98BADCFE, and 0x10325476) need to be the md5 hash of the 65th key (first 16 bytes). Is this correct?

    To do that I overrided the Initialize function of the MD5CryptoServiceProvider in my own MD5 class.

    public void Initialize(byte[] initializer)
    {
    count = 0;
    _ProcessingBufferCount = 0;

    _H[0] = initializer[0];
    _H[1] = initializer[1];
    _H[2] = initializer[2];
    _H[3] = initializer[3];
    }

    Thats all I changed. You can find the rest of the class here: http://www.java2s.com/Open-Source/CSharp/Database/Npgsql/Npgsql/MD5CryptoServiceProvider.cs.htm

    Then I rewrote my code to be this:

    private static byte[] getSectorIV(byte[] sector, long sectorIndex)
    {
    byte[] key65 = new byte[64];
    Array.Copy(getKey65IV(), 0, key65, 0, 16);

    byte[] sector496 = new byte[496];
    Array.Copy(sector, 16, sector496, 0, sector496.Length);

    byte[] sectorIndexBytes = new byte[7];
    Array.Copy(BitConverter.GetBytes(sectorIndex), 0, sectorIndexBytes, 0, sectorIndexBytes.Length);

    byte[] unhashedIV = new byte[sector496.Length + sectorIndexBytes.Length];

    Buffer.BlockCopy(sector496, 0, unhashedIV, 0, sector496.Length);
    Buffer.BlockCopy(sectorIndexBytes, 0, unhashedIV, sector496.Length, sectorIndexBytes.Length);

    //UInt32 bits = 4024;

    //Array.Copy(BitConverter.GetBytes(bits), 0, unhashedIV, unhashedIV.Length - 8, 4);

    MD5CryptoServiceProvider initialMD5 = new MD5CryptoServiceProvider();
    MyMD5 md5 = new MyMD5();

    md5.Initialize(initialMD5.ComputeHash(key65));

    return md5.ComputeHash(unhashedIV);
    }

    Something still isn't right though :-(

    My goal is for my utility to be contained to a single (Windows) executable. I didn't realize that the .NET AES functions were so distant from aespipe's implementation....but now that I'm so involved in this I'd hate to give up and start over :-)

    I really appreciate all the help you've been giving me though! Thanks!

     
  • Jari Ruusu
    Jari Ruusu
    2011-08-24

    Output of finalize() is not same as hash-internal state.
    finalize() adds padding (if necessary) and mixes in total
    number of input bits, and runs one more hash-internal
    transform. Internally MD5 processes data in 512 bit chunks.
    Those 48 bytes of zeroes are there so that it fills up one
    full 512 bit chunk. The state info that you want to preserve
    and copy is the state after one hash-internal transform.

    md5_initialize()
    md5_add_data(first 16 bytes of 65th key)
    md5_add_data(48 bytes of zeroes)
    saved_H0 = md5._H[0]
    saved_H1 = md5._H[1]
    saved_H2 = md5._H[2]
    saved_H3 = md5._H[3]
    md5_finalize()

    Above needs to be computed only once, and hash result can be
    ignored. You only care about those four 32 bit saved_H0 ...
    saved_H3 values.

    md5_initialize()
    md5._H[0] = saved_H0
    md5._H[1] = saved_H1
    md5._H[2] = saved_H2
    md5._H[3] = saved_H3
    md5_add_data(last 496 bytes of sector data)
    md5_add_data(7 bytes of sector number)
    md5_finalize()

    Input is 496 + 7 bytes, but hash computation starts with
    overridded internal state.

    OR alternatively, like this:

    md5_initialize()
    md5_add_data(first 16 bytes of 65th key)
    md5_add_data(48 bytes of zeroes)
    md5._total_length_bits = 0
    md5_add_data(last 496 bytes of sector data)
    md5_add_data(7 bytes of sector number)
    md5_finalize()

    Input is 16 + 48 + 496 + 7 bytes, but total length info is
    adjusted to ignore first 512 bits of input.

     
  • dan
    dan
    2011-08-24

    Okay, it feels like I'm getting closer, but still no luck decrypting :-/

    I added this function to my MyMD5 class to get the H values:

    public uint[] GetHValues()
    {
    uint[] HValues = new uint[4];

    HValues[0] = _H[0];
    HValues[1] = _H[1];
    HValues[2] = _H[2];
    HValues[3] = _H[3];

    return HValues;
    }

    And heres my new code to calculate the IVs. I know I only have to calculate the initial H values once (I'll move that later).

    private static byte[] getSectorIV(byte[] sector, long sectorIndex)
    {
    byte[] key65 = new byte[64];
    Array.Copy(getKey65IV(), 0, key65, 0, 16);

    byte[] sector496 = new byte[496];
    Array.Copy(sector, 16, sector496, 0, sector496.Length);

    byte[] sectorIndexBytes = new byte[7];
    Array.Copy(BitConverter.GetBytes(sectorIndex), 0, sectorIndexBytes, 0, sectorIndexBytes.Length);

    MyMD5 initialMD5 = new MyMD5();

    initialMD5.TransformFinalBlock(key65, 0, 64);

    MyMD5 md5 = new MyMD5();
    md5.Initialize(initialMD5.GetHValues());

    md5.TransformBlock(sector496, 0, sector496.Length, sector496, 0);
    md5.TransformFinalBlock(sectorIndexBytes, 0, sectorIndexBytes.Length);

    return md5.Hash;
    }

    I'm pretty confident this part of the code is doing what you say it should be doing now. Could the problem be somewhere else now?

    Thanks again!!
    -Dan

     
  • dan
    dan
    2011-08-25

    I think I may be having the same problem with the AES transform as I was with the MD5.

    I simplified my code as a test to see if I could encrypt/decrypt with only a single key. The file decrypted, however there was garbage in between each 512 byte sector.

    You had mentioned that each 512 byte block is encrypted independent of each other. I'm pretty sure the code I have to encrypt each sector is using some sort of TransformFinalBlock(). This probably adds padding before calculating the final transform. Is this what is supposed to happen or could this be another problem?

    Thanks again,
    Dan

     
  • Jari Ruusu
    Jari Ruusu
    2011-08-25

    dan-vri wrote:
    > I'm pretty confident this part of the code is doing what
    > you say it should be doing now.

    Nope.

    Please re-read my earier posts in this thread, and
    familiarize yourself about the difference of MD5 internal
    state _before_ finalize() vs. MD5 internal state _after_
    finalize().

    > I simplified my code as a test to see if I could
    > encrypt/decrypt with only a single key. The file
    > decrypted, however there was garbage in between each 512
    > byte sector.

    There is nothing _between_ 512 byte sectors. I assume that
    you meant garbage at beginning of 512 byte sector. Damaged
    data at beginning of 512 byte CBC-chain is a symptom of
    incorrect IV computation. Single-key mode 128 bit
    little-endian sector number IV is difficult to miscompute.

     
  • dan
    dan
    2011-08-25

    > Please re-read my earier posts in this thread, and
    > familiarize yourself about the difference of MD5 internal
    > state _before_ finalize() vs. MD5 internal state _after_
    > finalize().

    Okay, so my mistake was this:

    initialMD5.TransformFinalBlock(key65, 0, 64);
    MyMD5 md5 = new MyMD5();
    md5.Initialize(initialMD5.GetHValues());

    which needs to be this because I need the H values before it's finalized:

    initialMD5.TransformBlock(key65, 0, 64);
    MyMD5 md5 = new MyMD5();
    md5.Initialize(initialMD5.GetHValues());

    Is this right?

    > There is nothing _between_ 512 byte sectors. I assume that
    > you meant garbage at beginning of 512 byte sector. Damaged
    > data at beginning of 512 byte CBC-chain is a symptom of
    > incorrect IV computation. Single-key mode 128 bit
    > little-endian sector number IV is difficult to miscompute.

    For testing I've been using a file which contains nothing but 106,496 'A' characters

    This is my code to calculate the simple sector IVs:

    private static byte[] getSingleSectorIV(long sectorIndex)
    {
    byte[] sectorIndexBytes = new byte[16];
    Array.Copy(BitConverter.GetBytes(sectorIndex), 0, sectorIndexBytes, 0, BitConverter.GetBytes(sectorIndex).Length);

    Console.WriteLine(generateHashString(sectorIndexBytes));

    return sectorIndexBytes;
    }

    This produces IVs like this:

    0x00000000000000000000000000000000
    0x01000000000000000000000000000000
    ...
    0xCF000000000000000000000000000000

    Please take a look at the original and decrypted (single key) files I've uploaded. The first sector decrypts fine, but like you said the beginings of the other sectors aren't right. Theres also some random bytes in the middle of some sectors it seems.

    Thanks again!

     
  • dan
    dan
    2011-08-25

     
    Attachments
  • dan
    dan
    2011-08-25

     
    Attachments
    dec
  • Jari Ruusu
    Jari Ruusu
    2011-08-26

    I looked at your input and output file sizes. Input file
    size is 208*512 bytes. Output file size is 208*(512+16)
    bytes. So your program reads 208*512 byte sectors and
    intends to write 208*512 byte sectors. But you didn't even
    bother to check that output file size is expected size.

    Dan,
    I know this is not the most politically correct way of
    saying this, but currently your programming skill level is
    such that you need general purpose programming help and
    guidance. This loop-AES/aespipe support forum is wrong place
    for that. Debugging silly bugs in your code is off-topic
    here.

    I believe I have explained loop-AES/aespipe on-disk format
    with enough detail that other 100% compatible
    implementations can be written. If you have more
    loop-AES/aespipe on-disk format, usage, or compile related
    questions, then I will be happy to answer them.

     
  • dan
    dan
    2011-08-26

    > I looked at your input and output file sizes. Input file
    > size is 208*512 bytes. Output file size is 208*(512+16)
    > bytes. So your program reads 208*512 byte sectors and
    > intends to write 208*512 byte sectors. But you didn't even
    > bother to check that output file size is expected size.

    I did bother to check the file size. I had read before that AES encrypted files are sometimes larger than the original. I realize now that this is only because of padding and that if the original is a multiple of 16 bytes, the output should be the same size.

    > but currently your programming skill level is
    > such that you need general purpose programming help and
    > guidance. This loop-AES/aespipe support forum is wrong place
    > for that. Debugging silly bugs in your code is off-topic
    > here.

    I disagree. This was not a silly bug...on my part anyway. The .NET AES algorithm was adding a full additional 16 bytes of PKCS7 padding which is completely unneeded because the 512 byte sector is already a multiple of 16 bytes.

    Because it seems I'm going to have to check for the need of and add padding on my own (for the last sector), can you confirm that aespipe expects null bytes as padding?

    > I believe I have explained loop-AES/aespipe on-disk format
    > with enough detail that other 100% compatible
    > implementations can be written. If you have more
    > loop-AES/aespipe on-disk format, usage, or compile related
    > questions, then I will be happy to answer them.

    Thanks for helping me discover this problem, the files are decrypting fine now (single and v3).

     
  • Jari Ruusu
    Jari Ruusu
    2011-08-27

    dan-vri wrote:
    > I did bother to check the file size.

    I doubt that. At least you never compared your ciphertext
    size to aespipe encrypted ciphertext size.

    There was no mention of any padding, because 512-byte CBC
    chains do not need any padding. "CBC-chains are 512 bytes
    long" really meant 512 byte length, not 528 byte length.
    Original tool (aespipe) does not expand encrypted output
    data size at all if plaintext input data size is multiple of
    512 bytes. Buggy implementation does (or did) expand
    encrypted output data size. These facts should have been
    enough so that a skilled coder be able to figure out the
    problem without asking anyone else for help.

    Another big clue about aespipe encrypted data fitting into
    same number of 512-byte sectors as plaintext data is
    loop-AES Linux kernel driver, which is a transparent way to
    encrypt normal file systems. Linux kernel file system
    (ext2/ext3/ext4/xfs/reiserfs/whatever) sends
    multiple-of-512-bytes sized plaintext write or read requests
    to loop-AES driver, which sends multiple-of-512-bytes sized
    write or read requests to backing device driver, encrypting
    or decrypting the data in the process. One of the examples
    in aespipe README file (section 3.4.) even explains how to
    encrypt or decrypt a file system in-place. That would not
    work at all if encrypted data size was larger than plaintext
    data size.

    > can you confirm that aespipe expects null bytes as
    > padding?

    aespipe fills partial sector data with null bytes.

     
  • Jari Ruusu
    Jari Ruusu
    2011-09-06

    • status: open --> closed