[Lame-dev] [ lame-Bugs-1227632 ] Possible assertion failure in takehiro.c::count_bits()

[SourceForge.net <no...@so...>]

Bugs item #1227632, was opened at 2005-06-26 00:58
Message generated for change (Comment added) made by pa3pyx
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=100290&aid=1227632&group_id=290

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Libraries
Group: Compatibility
Status: Open
Resolution: None
Priority: 5
Submitted By: Pa3PyX (pa3pyx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Possible assertion failure in takehiro.c::count_bits()

Initial Comment:
If for some reason calc_min_bits() or on_pe() (for CBR) 
returns too few bits for granule, we could have 
outer_loop() set global gain of 256. If balance_noise() 
also happens to enable scalefac_scale, then we have 
cod_info->global_gain + cod_info->scalefac_scale == 
257. Of course this means a failed quantization and if 
debugging is disabled, VBR routine will simply retry with 
more bits. If not, however, we get an assertion failure in 
count_bits() attempting out of array access on ipow20.

Granted that this is very unlikely and does not affect 
quality -- however, there appears to be nothing in the 
code to prevent this from happening.

Solution: test global_gain < 256 *before* doing 
count_bits() in inner loop. Theoretically, this could also 
happen in bin_search_StepSize() (last else if), but 
practically I never had assert(global_gain < 256) fail.

Side note: I only had this happen after tweaking the bit 
allocation code for CBR to use VBR_encode_granule 
routine (min_bits being max(0, (mean_bits - req'd to refill 
reservoir) * channel ratio, max_bits being 
min(max_bits(on_pe) * channel ratio, 4095)). Coupled 
with noise based (quant_compare) MS/LR decision 
(instead of fixed PE based), this seems to use bit 
reservoir more actively and provide somewhat higher 
quality for medium bitrate CBR, although brings in all the 
VBR glitches that result from excessive reliance on the 
psymodel...

----------------------------------------------------------------------

>Comment By: Pa3PyX (pa3pyx)
Date: 2005-06-26 01:00

Message:
Logged In: YES 
user_id=499949

Addition: Verified that this can happen with 126 bits per 
granule (minimum allocated by VBR_prepare()).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=100290&aid=1227632&group_id=290