Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#213 Wav header incorrect when 4 GB file size exceeded

open
nobody
None
5
2014-08-05
2012-08-15
Mikael Hartzell
No

Bug: Wav file header gets incorrect information when 4 GB file size is exceeded
--------------------------------------------------------------------------------------------------------------------

OS: Ubuntu 12.04
Version: Sox version v14.4.2 built from latest git commit (dated 2012.08.07): 35eebaf602ffd1477da1b604d345fab37aa8d42c

How to reproduce bug
---------------------

- Create two 6 channel 48 kHz, 16 bit audio files, one with duration of 2 hours 4 minutes and the other with 2 hours and 5 minutes.
- The latter will exeed 4 GB barrier and wav header file duration and sample count information gets mixed up.
- Get file duration from both files with sox --i and ffprobe (is included in FFmpeg).
- Sox --i will tell file durations are: 02:04:00.00 and 00:00:43.46 though they should both be 02:04:00.00. Also sample count is from in latter file.
- ffprobe will tell both files are 02:04:00.00, I don't know how it gets the information correctly since sox --i does not.
- VLC will only play 00:00:43.46 of the latter file.
- The wav header gets mixed up also when converting from flac to wav with sox if the file size exceeds 4 GB.

Commands to reproduce the bug
-----------------------------

- Create test files by running both of these commands:

sox -V -V -n -b 16 testfile-duration_124min_6_channels.wav trim 0 02:04:00.00 remix 0 0 0 0 0 0
sox -V -V -n -b 16 testfile-duration_125min_6_channels.wav trim 0 02:05:00.00 remix 0 0 0 0 0 0

- Check with ls -l that the file size of both files seems to be correct.
- Use sox --i on both files to get file size.
- Use ffprobe on both files to get file size.

Mikael Hartzell

Text output from my test. Shell commands are underlined:

sox -V -V -n -b 16 testfile-duration_124min_6_channels.wav trim 0 02:04:00.00 remix 0 0 0 0 0 0 ; sox -V -V -n -b 16 testfile-duration_125min_6_channels.wav trim 0 02:05:00.00 remix 0 0 0 0 0 0
----------------------------------------------------------------------------------------------------------------------
sox: SoX v14.4.2
time: Aug 15 2012 20:58:12
uname: Linux mika-HP-Compaq-nc6320-EV073AV 3.2.0-29-generic-pae #46-Ubuntu SMP Fri Jul 27 17:25:43 UTC 2012 i686
compiler: gcc 4.6.3
arch: 1248 48 44 L OMP
sox INFO nulfile: sample rate not specified; using 48000

Input File : '' (null)
Channels : 1
Sample Rate : 48000
Precision : 32-bit

sox DBUG wav: Writing Wave file: Microsoft PCM format, 6 channels, 48000 samp/sec
sox DBUG wav: 576000 byte/sec, 12 block align, 16 bits/samp

Output File : 'testfile-duration_124min_6_channels.wav'
Channels : 6
Sample Rate : 48000
Precision : 16-bit
Sample Encoding: 16-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

sox DBUG remix: 0:
sox DBUG remix: 1:
sox DBUG remix: 2:
sox DBUG remix: 3:
sox DBUG remix: 4:
sox DBUG remix: 5:
sox INFO sox: effects chain: input 48000Hz 1 channels (multi) 32 bits unknown length
sox INFO sox: effects chain: trim 48000Hz 1 channels (multi) 32 bits 02:04:00.00
sox INFO sox: effects chain: remix 48000Hz 6 channels (multi) 32 bits 02:04:00.00
sox INFO sox: effects chain: dither 48000Hz 6 channels 16 bits 02:04:00.00
sox INFO sox: effects chain: output 48000Hz 6 channels (multi) 16 bits 02:04:00.00
sox DBUG sox: start-up time = 0.001637
sox DBUG input: output buffer still held 2048 samples; dropped.
sox DBUG wav: Finished writing Wave file, 4285440000 data bytes 357120000 samples

sox: SoX v14.4.2
time: Aug 15 2012 20:58:12
uname: Linux mika-HP-Compaq-nc6320-EV073AV 3.2.0-29-generic-pae #46-Ubuntu SMP Fri Jul 27 17:25:43 UTC 2012 i686
compiler: gcc 4.6.3
arch: 1248 48 44 L OMP
sox INFO nulfile: sample rate not specified; using 48000

Input File : '' (null)
Channels : 1
Sample Rate : 48000
Precision : 32-bit

sox DBUG wav: Writing Wave file: Microsoft PCM format, 6 channels, 48000 samp/sec
sox DBUG wav: 576000 byte/sec, 12 block align, 16 bits/samp

Output File : 'testfile-duration_125min_6_channels.wav'
Channels : 6
Sample Rate : 48000
Precision : 16-bit
Sample Encoding: 16-bit Signed Integer PCM
Endian Type : little
Reverse Nibbles: no
Reverse Bits : no
Comment : 'Processed by SoX'

sox DBUG remix: 0:
sox DBUG remix: 1:
sox DBUG remix: 2:
sox DBUG remix: 3:
sox DBUG remix: 4:
sox DBUG remix: 5:
sox INFO sox: effects chain: input 48000Hz 1 channels (multi) 32 bits unknown length
sox INFO sox: effects chain: trim 48000Hz 1 channels (multi) 32 bits 02:05:00.00
sox INFO sox: effects chain: remix 48000Hz 6 channels (multi) 32 bits 02:05:00.00
sox INFO sox: effects chain: dither 48000Hz 6 channels 16 bits 02:05:00.00
sox INFO sox: effects chain: output 48000Hz 6 channels (multi) 16 bits 02:05:00.00
sox DBUG sox: start-up time = 0.001569
sox DBUG input: output buffer still held 5632 samples; dropped.
sox DBUG wav: Finished writing Wave file, 25032704 data bytes 360000000 samples

ls -l testfile-duration_12*
----------------------------
-rw-rw-r-- 1 mika mika 4285440080 elo 15 21:10 testfile-duration_124min_6_channels.wav
-rw-rw-r-- 1 mika mika 4320000080 elo 15 21:15 testfile-duration_125min_6_channels.wav

sox --i testfile-duration_12*
-----------------------------

Input File : 'testfile-duration_124min_6_channels.wav'
Channels : 6
Sample Rate : 48000
Precision : 16-bit
Duration : 02:04:00.00 = 357120000 samples ~ 558000 CDDA sectors
File Size : 4.29G
Bit Rate : 4.61M
Sample Encoding: 16-bit Signed Integer PCM

Input File : 'testfile-duration_125min_6_channels.wav'
Channels : 6
Sample Rate : 48000
Precision : 16-bit
Duration : 00:00:43.46 = 2086058 samples ~ 3259.47 CDDA sectors
File Size : 4.32G
Bit Rate : 795M
Sample Encoding: 16-bit Signed Integer PCM

Total Duration of 2 files: 02:04:43.46

ffprobe testfile-duration_124min_6_channels.wav
------------------------------------------------
avprobe version 0.8.3-4:0.8.3-0ubuntu0.12.04.1, Copyright (c) 2007-2012 the Libav developers
built on Jun 12 2012 16:37:58 with gcc 4.6.3
[wav @ 0x92e4aa0] max_analyze_duration reached
Input #0, wav, from 'testfile-duration_124min_6_channels.wav':
Duration: 02:04:00.00, bitrate: 4608 kb/s
Stream #0.0: Audio: pcm_s16le, 48000 Hz, 5.1, s16, 4608 kb/s

ffprobe testfile-duration_125min_6_channels.wav
------------------------------------------------
avprobe version 0.8.3-4:0.8.3-0ubuntu0.12.04.1, Copyright (c) 2007-2012 the Libav developers
built on Jun 12 2012 16:37:58 with gcc 4.6.3
[wav @ 0x8555aa0] max_analyze_duration reached
Input #0, wav, from 'testfile-duration_125min_6_channels.wav':
Duration: 02:05:00.00, bitrate: 4608 kb/s
Stream #0.0: Audio: pcm_s16le, 48000 Hz, 5.1, s16, 4608 kb/s

Discussion

  • typos creaped in the bug report, corrected lines:

    - Sox --i will tell file durations are: 02:04:00.00 and 00:00:43.46 though they should be 02:04:00.00 and 02:05:00.00. Also sample count is wrong in latter file.
    - ffprobe will tell files are 02:04:00.00 and 02:05:00.00, I don't know how it gets the information correctly since sox --i does not.

     
  • Ulrich Klauer
    Ulrich Klauer
    2012-08-30

    Well, there is no way to do this right. RIFF/WAVE only uses 32 bits to store lengths, so anything >4 GiB won't fit. This is comparable to a paper form that asks for your age, but only offerss two places for the digits; if you're 102 years old, you might write "02" or "99", but you can't specify your true age.

    We should perhaps choose the equivalent of "99". But the only way to completely avoid the problem is for you to switch to a different file format like W64 or FLAC.

     
  • Hi :)

    Thanks for replying :)

    Yes I am aware that the 'old' wav format has a hard limit of 4 GB, but the way sox handles this limit is in my opinion a little troublesome.

    Imagine a scenario where you convert a 8 channel flac file to wav. It is very possible that the output file size may exceed wav format max limit of 4 GB. Sox does not enforce the limit but instead lets the header become invalid. Most of the data in the file becomes not playable in standard players. When sox is used in a script to automate file conversion, this error goes unnoticed until somebody tries to open and play the file.

    May I humbly suggest two possible workarounds :)

    1. When writing wav output sox checks for the 4 GB limit, and stops copying any more data to the ouput file when the limit is reached. Sox should display an error message informing the user that the output file was truncated.

    2. When writing wav output sox checks for the 4 GB limit and when it is crossed sox writes a w64 header to the file and continues putting more data to the output file.

    In my opinion it is very important to keep the header in a sane condition and in sync with the data in the file. Suggestion 1 would be fairly easy to implement and should prevent creating invalid files without the user noticing.

    I built another open source project that uses your program and I have just now become to know sox better and I am very impressed what all it can do :) So Thanks guys for your great work :)

     
  • Hi again :)

    I was thinking about the case where a user converts a compressed multichannel audio file to uncompressed wav and I had an idea :)

    The easiest and most efficient way to solve this problem would be to not let sox do any work if it can see that the output file size would go over 4 GB.

    When sox starts, it propably has all the information about the input file it needs and can estimate the uncompressed output file size for example using sample count, bit depth and channel count. Sox could then output an error message and refuse to do the conversion if the resulting data can not be stored in a wav container without breaking the header.

    I think this solution might be quite easy to implement and could be used in all cases where the output file is a 32 bit container.

    Mikael Hartzell