Out of bound array access in twolame_bits_for_nonoise()

[svart-riddare]

The problem was seen by running an ASAN (address sanitizer) build. Not exactly sure about the fix.

[attritionorg]

@svart-riddare Could you post the ASAN output please?

[svart-riddare]

Here is the (truncated, I have not included application stack traces) ASAN output :

=================================================================
==219322==ERROR: AddressSanitizer: global-buffer-overflow on address 0x00001816457c at pc 0x00000b1b110b bp 0x7fb3f7719b60 sp 0x7fb3f7719b50
READ of size 4 at 0x00001816457c thread T9
    #0 0xb1b110a in twolame_bits_for_nonoise /twolame-0.4.0/libtwolame/encode.c:754
    #1 0xb1b3e6d in twolame_main_bit_allocation /twolame-0.4.0/libtwolame/encode.c:922
    #2 0xb1a32b1 in encode_frame /twolame-0.4.0/libtwolame/twolame.c:602
    #3 0xb1a7172 in twolame_encode_buffer_float32_interleaved /twolame-0.4.0/libtwolame/twolame.c:927

0x00001816457c is located 4 bytes to the left of global variable 'nbal' defined in '/twolame-0.4.0/libtwolame/encode.c:89:18' (0x18164580) of size 36
0x00001816457c is located 52 bytes to the right of global variable 'steps' defined in '/twolame-0.4.0/libtwolame/encode.c:94:18' (0x18164500) of size 72
SUMMARY: AddressSanitizer: global-buffer-overflow /twolame-0.4.0/libtwolame/encode.c:754 in twolame_bits_for_nonoise
Shadow bytes around the buggy address:
  0x000083024850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x000083024860: 00 00 00 00 f9 f9 f9 f9 00 00 04 f9 f9 f9 f9 f9
  0x000083024870: 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9
  0x000083024880: 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9
  0x000083024890: 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9
=>0x0000830248a0: 00 00 00 00 00 00 00 00 00 f9 f9 f9 f9 f9 f9[f9]
  0x0000830248b0: 00 00 00 00 04 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0000830248c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000830248d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000830248e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000830248f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc

==219322==ABORTING

With the debugger we then found that the code was sometimes reading array nbal[-1]; we thought it was probably a bad idea, hence our commit.

[eblanca]

Both table number and sblimit are selected in order to get meaningful values between 0 and 8 (look at the contents of array 'line'). Values -1 always lie beyond the sblimit bound.
If -1 gets selected from 'line', then something has happened before, which is causing table and/or sblimit to be wrongly picked.
Can you someway share the audio content (and the used command line) causing this behaviour ?

[svart-riddare]

Eighteen months lather, I'll see if I have around the original material that shows the issue.

I remember though that it was related to the encoding mode TWOLAME_JOINT_STEREO.