Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

corrupt mxf files

Help
2009-06-10
2013-04-25
1 2 > >> (Page 1 of 2)
  • Ilter Cimilli
    Ilter Cimilli
    2009-06-10

    Anyone have any Idea's on how to recover any available video frames from a recording that crashed during capture (power loss).
    I have several MXF files 4,5GB that will not play or import on anything. I guess the file is not closed off or that the header is incomplete. I do not know enough about the mxf format yet but someone might have some idea's to complete repair or extract  data from these files?
    Ilter.

     
    • Sean Casey
      Sean Casey
      2009-06-11

      Hi Ilter,

      You're correct - the header data is written at the end of the recording process and the files are then moved from the 'creating' directory to the correct location. If the computer is turned off before this is done, the only way to fix these files is to rewrite the header data. Luckily there are tools available to do this.

      As it happens I'm just working on recovering some files which suffered a similar fate. I'll get back to you with instructions on how to do this soon.

      -Sean

       
      • Ilter Cimilli
        Ilter Cimilli
        2009-06-18

        Any clues on where to look for existing tools or utilities?
        just about every mxf repair tool seems to be specifically for DV resolutions and the P2 world. Where as I need something to recognize Avid MJPEG 2:1,
        I don't need sound, or even TC...
        Is the process something like - using a hex editor to copy and paste from a known "good header file" and then somehow work out the duration discrepancies? Or is that oversimplifying?

        Ilter.

         
        • Hi, I've just finished recovering the files I mentioned. There are a few steps to recreating the broken mxf file:

          1) Extract the raw essence data from the incomplete mxf file by extracting all data after a certain offset:
          dd ibs=1 if=infile.mxf of=out.raw skip=40100

          2) Run writeavidmxf to recreate the mxf file (you can specify various additional parameters in this, such as timecode if you require):
          ap_workspace/ingex/libMXF/examples/writeavidmxf/writeavidmxf --prefix out.mxf --project myproject --mjpeg out.raw --res "2:1"

          3) Test
          player -m out.mxf

          [In the above replace infile.mxf with your incomplete mxf filename]

          This should create a finalised file called out.mxf. Let me know if you have any trouble with this!

          Cheers,
          -Sean

           
          • Sean Casey
            Sean Casey
            2009-06-19

            Correction to my previous reply: the 'skip' parameter on the first command should be '262400', ie:

            dd ibs=1 if=infile.mxf of=out.raw skip=262400

            Also if you want to include the audio tracks, these need to be extracted as raw data using the above command on each audio file, then specified using --pcm parameters on the writeavidmxf command, for example:

            ap_workspace/ingex/libMXF/examples/writeavidmxf/writeavidmxf --prefix out.mxf --project myproject --mjpeg out.raw --res "2:1" --pcm audio1.mxf --pcm audio2.mxf

            [presumes 2 audio tracks named audio1.mxf and audio2.mxf]

            -Sean

             
    • Phil Tudor
      Phil Tudor
      2009-06-19

      Yes, offset is 262400 decimal = 0x40100 hex.

      If your clips had sound, you should extract *all* the tracks you want to recreate in the regenerated clip (e.g. v1, a1, a2), and pass all of these raw files to writeavidmxf in one go. This is because the clip metadata describing all the tracks is written into each track file.

      The command would be something like:
      writeavidmxf --prefix out.mxf --mjpeg v1.raw --pcm a1.raw --pcm a2.raw

      The argument to the --pcm option would be the raw pcm audio samples extracted using dd.

      Finally, if you want to recreate  the metadata from the original MXF file (e.g. material IDs, timecode, timestamps) in the recreated MXF files, use avidmxfinfo to display a summary of the key metadata and pass in the appropriate values to writeavidmxf.

      Phil

       
    • Ilter Cimilli
      Ilter Cimilli
      2009-06-22

      Thanks Sean & Phil but I must be doing something wrong here... so I've gone back to beginnings with little luck again, where am I going wrong?
      I could not get the process to work with the corrupt files so I started again from a known (good and working) mxf file
      the file is...

      Filename = /video/test/20090620_18275223_Test_2009_Ingex_input1_0_1_v1.mxf
      Project name = Test 2009
      Project edit rate = 25/1
      Clip name = 20090620_18275223_Test_2009_Ingex_input1_0
      Clip created = 2009-06-20 08:27:56.000
      Clip edit rate = 25/1
      Clip duration = 430 samples (00:00:17:05)
      Clip video tracks = 1
      Clip audio tracks = 0
      Clip track string = V1
      Video essence
      Essence type = 2:1
      Essence label = 060e2b34040101010e04030102010000
      Track number = 1
      Edit rate = 25/1
      Duration = 430 samples (00:00:17:05)
      Start timecode = 1661823 samples (18:27:52:23)
      Image aspect ratio = 16/9
      Stored WxH = 720x296 (separate fields)
      Display WxH = 720x296 (separate fields)
      User comments:
        Shoot Date = 20090620
        Descript = Test Rec
        Location = Test
      Material package attributes:
        _PJ = Test 2009
      Material package UID = 060c2b340205110101041000130000004a3c9d8caa1a6668060e2b347  f7f2a80
      File package UID     = 060c2b340205110101041000130000004a3c9d8caa1b6668060e2b347  f7f2a80
      Physical package UID = 060c2b340205110101041000130000004a3c9cbb58a76668060e2b347  f7f2a80
      Physical package type = Tape
      Physical package name = W095

      *************Take this file and then...

      dd ibs=1 if=20090620_18275223_Test_2009_Ingex_input1_0_1_v1.mxf of=out40100.raw skip=40100
      24082365+0 records in
      47035+1 records out
      24082365 bytes (24 MB) copied, 14.9883 s, 1.6 MB/s

      *************or incase wrong offset...

      dd ibs=1 if=20090620_18275223_Test_2009_Ingex_input1_0_1_v1.mxf of=out262400.raw skip=262400
      23860065+0 records in
      46601+1 records out
      23860065 bytes (24 MB) copied, 14.6886 s, 1.6 MB/s

      *************now pass the files through the mxfwriter...

      /home/ingex/ap-workspace/ingex/libMXF/examples/writeavidmxf/writeavidmxf --prefix /video/test/out40100.mxf --project Test 2009 --start-tc 17:52:12:04 --tape input0 --mjpeg /video/test/out40100.raw --res "2:1"
      Warning: MJPEG image start is non-0xFF byte - trailing data ignored
      Warning: near file offset 8192

      *************or with the alternative offset...

      /home/ingex/ap-workspace/ingex/libMXF/examples/writeavidmxf/writeavidmxf --prefix /video/test/out262400.mxf --project Test 2009 --start-tc 17:52:12:04 --tape input0 --mjpeg /video/test/out262400.raw --res "2:1"
      Warning: MJPEG image start is non-0xFF byte - trailing data ignored
      Warning: near file offset 8192

      *************in both cases the resulting mxf file is N.G. and tiny (384.6 KB)

      what does the warning  "MJPEG image start is non-0xFF byte - trailing data ignored" mean?
      Ilter.

       
      • Sean Casey
        Sean Casey
        2009-06-22

        Ok, it seems like the video data is starting in a different place than expected. Could you run the following command on the original mxf video file:

        od -A x -t -x1 original_file.mxf | less

        ...scroll to the line which begins '040100' and let me know the data contained on the line?

        Also the video seems very short (17 secs) - is this correct?

        -Sean

         
    • Ilter Cimilli
      Ilter Cimilli
      2009-06-23

      That's right this is a short test file that actualy works (is not corrupt).
      I figured until I can extract the essance data and re-wrap from a known good file there would be liitle point trying again on the corrupt ones.

      0400a0 00 06 0e 2b 34 01 01 01 01 03 01 02 10 01 00 00
      0400b0 00 88 00 00 00 00 00 00 00 2d 00 00 00 00 00 00
      0400c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      *
      0400e0 00 00 00 00 00 00 00 06 0e 2b 34 01 02 01 01 0e
      0400f0 04 03 01 15 01 01 01 87 00 00 00 01 6b 3c 65 ff
      040100 d8 ff e0 00 10 41 56 49 31 00 00 00 00 6b c0 00
      040110 00 6b c0 ff fe 00 3d 41 56 49 44 11 00 00 00 00
      040120 00 00 4c 02 00 00 00 00 00 00 00 00 00 00 00 00
      040130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      *
      040150 00 00 ff dd 00 04 00 00 ff db 00 84 00 01 01 01
      040160 01 01 01 01 01 01 01 01 01 01 01 01 02 01 01 01
      040170 01 01 02 01 01 01 02 02 02 02 02 02 02 02 02 03

      Ilter.

       
      • Sean Casey
        Sean Casey
        2009-06-24

        Hi, looking at that data, it seems the video offset is 1 byte different to the files I have been working with. Could you do the same for the broken mxf file, so I can give you the correct command for extracting the data?

        -Sean

         
    • Ilter Cimilli
      Ilter Cimilli
      2009-06-24

      ok here's one of the broken fiiles...
      *
      040070 00 00 00 00 00 00 00 00 01 06 0e 2b 34 04 01 01
      040080 02 0d 01 02 01 10 02 00 00 00 00 00 01 00 00 00
      040090 10 06 0e 2b 34 04 01 01 01 0e 04 03 01 02 01 00
      0400a0 00 06 0e 2b 34 01 01 01 01 03 01 02 10 01 00 00
      0400b0 00 88 00 00 00 00 00 00 00 2d 00 00 00 00 00 00
      0400c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      *
      0400e0 00 00 00 00 00 00 00 06 0e 2b 34 01 02 01 01 0e
      0400f0 04 03 01 15 01 01 01 87 00 00 00 00 00 00 00 ff
      040100 d8 ff e0 00 10 41 56 49 31 00 00 00 02 a4 14 00
      040110 02 a4 14 ff fe 00 3d 41 56 49 44 11 00 00 00 00
      040120 00 00 4c 02 00 00 00 00 00 00 00 00 00 00 00 00
      040130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      *
      040150 00 00 ff dd 00 04 00 00 ff db 00 84 00 01 01 01
      040160 01 01 01 01 01 01 01 01 01 01 01 01 02 01 01 01
      040170 01 01 02 01 01 01 02 02 02 02 02 02 02 02 02 03
      040180 03 04 03 03 03 03 03 02 02 03 04 03 03 04 04 04
      040190 04 04 02 03 05 05 04 04 05 04 04 04 04 01 01 01
      0401a0 01 01 01 01 02 01 01 02 04 03 02 03 04 04 04 04
      0401b0 04 04 04 04 04 04 04 04 04 04 04 04 04 04 04 04
      *

      Ilter.

       
      • Sean Casey
        Sean Casey
        2009-06-24

        Ok, looks like it has the same offset as the broken one, so if you use the following command instead to extract the raw video data:

        dd ibs=1 if=infile.mxf of=out.raw skip=400ff

        ...and then run writemxf as before.

        -Sean

         
        • Phil Tudor
          Phil Tudor
          2009-06-24

          Remember that the parameters to dd are in decimal. One byte earlier would be "skip=262399".

          Phil

           
          • Sean Casey
            Sean Casey
            2009-06-24

            Yes, my mistake :o) trying to do too many things at once!

            dd ibs=1 if=infile.mxf of=out.raw skip=262399

            -Sean

             
    • Ilter Cimilli
      Ilter Cimilli
      2009-06-26

      WOW :) All worked like a charm....Thank you!
      What was the string you were looking for within the data (ff d8 ff e0 00 10)?
      Many MANY Thanks.
      Ilter.

       
      • Sean Casey
        Sean Casey
        2009-06-30

        Great :) glad you got it sorted!

        Yes the hex values you quote represent the beginning of the raw video data.

        -Sean

         
  • stefano
    stefano
    2010-12-13

    Hi,
    I have this problem:
    I have folder Creating with many files.
    When I go this command "od -A x -t -x 1 original_file.mxf | less"
    I don't have the row 040100 but :

    0400a0 00 06 0e 2b 34 01 01 01 01 03 01 02 10 01 00 00
    0400b0 00 88 00 00 00 00 00 01 ff 2d 00 00 00 00 00 00
    0400c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    *
    05ffe0 00 00 00 00 00 00 00 06 0e 2b 34 01 02 01 01 0e
    05fff0 04 03 01 15 01 01 01 88 00 00 00 00 00 00 00 00
    060000 ff d8 ff e0 00 10 41 56 49 31 00 00 00 03 8d 71
    060010 00 03 8d 71 ff fe 00 3d 41 56 49 44 11 00 00 00
    060020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    *
    060050 00 00 00 ff dd 00 04 00 00 ff db 00 84 00 01 01
    060060 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
    060070 01 01 01 01 01 01 01 01 01 01 01 02 01 01 01 01
    060080 01 01 01 01 01 01 01 03 02 02 03 01 01 01 01 01

    I did not understand the offset.
    I can re-explain?
    thanks
    stefano

     
  • roland35
    roland35
    2010-12-16

    Hi,

    I have exactly the same problem as stefano - there is no row 040100, in all recorded files. Is there a way to find out the beginning of the essence container?

    Roland

     
  • Philip de Nier
    Philip de Nier
    2010-12-16

    The essence container in Ingex Avid MXF files start at offset 0x5ffe7 for video essence and offset 0x40fe7 for audio essence. The essence data starts 16+9 bytes after that offset, ie. at 0x60000 for video and 0x41000 for audio.

    The essence is clip wrapped and therefore the essence container starts with a key corresponding to the essence type  (eg. 0x060e2b34010201010e04030115010101 for MJPEG) followed by a 1+8-byte BER length (eg. 0x880000000000000000 in example above tells me the essence wasn't completed - size wasn't updated from 0) and then followed by the essence data.

    Another way to locate the essence container is to locate the body partition pack instead; first 14 bytes of the key are 0x060e2b34020501010d0102010103). After the partition pack you will probably find a fill element (0x060e2b34010101010301021001000000 at offset 0x0400a1 in the example above) followed by the essence element key.

    Philip

     
  • roland35
    roland35
    2010-12-16

    thank you Philip, I tried to unwrap an "good" file with dd and to wrap it again with writeavidmxf, just for the case of emergency. With your information I've got it working. Ingex player can play it, unfortunately MC's Media Tool ignores it. I wrapped it with the following command line: writeavidmxf -prefix testout2.mxf -project Ingex3 -start-tc 05:10:32:05 -mjpeg testout2.raw -res "2:1"
    By the way, is there a tool for wrapping OP-1A files, too?

    Roland

     
  • Philip de Nier
    Philip de Nier
    2010-12-16

    Another option (for Ingex Avid MXF files) is the recovery software in ingex/studio/processing/recover_mxf which uses OPAtomReader in libMXF++/examples/OPAtomReader

    MC shouldn't be ignoring the file. Ingex Studio is using the same software that the writeavidmxf commandline utility is using. Has Avid updated it's database? (Also, I'd use "-prefix testout2" because writeavidmxf creates files with names <prefix>_v1.mxf, <prefix_a1.mxf etc.)

    We don't have a general purpose OP-1A writer. There is an IMX OP-1A writer.

    Philip

     
  • roland35
    roland35
    2010-12-20

    Hi,

    only to complete the picture, MC certainly is not ignoring the writeavidmxf-wrapped files, provided they are in the correct directory - sorry.

    Roland

     

  • Anonymous
    2011-05-03

    Hi,
    I've just recently step into an accidentally formatted P2 card. I've been able to retrieve all the data but it seems the MXF flies got corrupted or scrambled in the process. This is the only thread that gets close to fixing this files successfully.
    I was able to create a .raw file out of the corrupted .mxf using dd, without any luck on wrapping it again to mxf format.

    I'm getting the error:

    Warning: MJPEG image start is non-0xFF byte - trailing data ignored

    Warning: near file offset 8192

    I've tried to keep up with the resolution Sean addressed in here, but it seems the offset of my data file is elsewhere, other than 0x40100 or 0x400ff. Honestly I don't really understand how to identify the offset of the MXF file. I do know how to transform de hex value into decimal for the "skip" attribute, but I don't really know where to get the hex value from.

    For some reason, I can't get the od command to work, so I had to download a GUI tool called HexEditor to get the following values:

    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    5f 77 00 60 ff ff d8 7f  61 3f ca fc ff 62 ff 1f
    ff ff 63 18 04 34 08 64  cb 24 0f ab ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    5f 77 01 ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    5f 77 02 ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
    7f 77 00 50 56 11 c3 c0  fe a4 fd 5e fe ca ff f4        <---- 40100 line
    02 15 01 b8 00 6a ff 23  fd 46 fe 86 00 a4 02 06
    02 6c ff fc fe 6f fd 70  fe ed 01 85 02 09 01 72
    ff 95 fe 83 fd f7 ff fd  00 b1 02 36 00 a1 fe a4
    ff e6 fe 84 00 0f 01 02  02 05 00 ee fe 48 fd 9f
    9f 77 00 06 6b 0c 1c 16  ee 40 41 2f a0 8a 6b 46
    59 af e5 5b 36 f6 bd 5f  6b c1 e1 37 44 7f 41 ed
    7f 39 6b 4e 13 3b d1 1c  fa fa bb a2 f8 c6 ec bd
    60 8a 29 7c 03 cb f8 c6  db cc bf b6 fb bf c7 be
    ff 46 b9 1f c9 79 bf a2  ff 46 bb 11 fe 58 ed 09
    9f 77 01 06 66 0f 15 8a  09 df c9 d5 82 ce 66 46
    3a f8 00 08 0b f4 4b 4f  65 cc 7c 06 fa 23

    Any help would be appreciated guys. Thank you forehand.

    Cristyan J. Moreno
    http://about.me/cjmoreno

     
  • Philip de Nier
    Philip de Nier
    2011-05-03

    Is it a P2 file or an Avid file?

    Philip

     

  • Anonymous
    2011-05-04

    Hi Philip it's a P2 file, but no worries about file structure of P2's card, I just need to fix the corrupted MXF files.

    I can open non-corrupted MXF files with my Quicktime + (Calibrated plugin). If I can do that with my corrupted files then I'll be saved.

    I know files can be fixed, before trying all this, I stumbled upon a recovery tool called Treasured (http://aeroquartet.com/movierepair/repair.html). I was able to watch several chunks of all my videos but they ask for money so you can successfully save them.

    Thank you. Regards.

    Cristyan J. Moreno
    http://about.me/cjmoreno

     
1 2 > >> (Page 1 of 2)