[Opalvoip-devel] H264 codec failed for latest Luyten svn

[Alexander S. <ale...@gm...>]

   Up-to-date H264 codec show that something messing with a memory. This bug
is not clearly visible at all times. "simpleopal" fails with H264
codec,"codectest" works fine. 

Normally it manifest itself as a memory allocator corruption during
"simpleopal" invocation:

*** glibc detected *** obj_linux_x86_64/simpleopal: free(): invalid next
size (normal): 0x0000000002439320 ***

======= Backtrace: =========

/lib/x86_64-linux-gnu/libc.so.6(+0x72606)[0x7fe965410606]

/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fe96541533c]

/usr/local/lib/libpt.so.2.10-beta2(_ZN14PAbstractArray15DestroyContentsEv+0x
35)[0x7fe96611a81d]

/usr/local/lib/libpt.so.2.10-beta2(_ZN10PContainer14AssignContentsERKS_+0x84
)[0x7fe96611ab7e]

obj_linux_x86_64/simpleopal(_ZN14PAbstractArray14AssignContentsERK10PContain
er+0x19)[0x4187c9]

/usr/local/lib/libopal.so.3.10-beta2(_ZN14PAbstractArrayaSERKS_+0x2e)[0x7fe9
66a37a90]

/usr/local/lib/libopal.so.3.10-beta2(_ZN10PBaseArrayIhEaSERKS0_+0x23)[0x7fe9
66a38ec7]

/usr/local/lib/libopal.so.3.10-beta2(_ZN10PBYTEArrayaSERKS_+0x23)[0x7fe966a3
8ef1]

/usr/local/lib/libopal.so.3.10-beta2(_ZN13RTP_DataFrameaSERKS_+0x23)[0x7fe96
6a6e01b]

/usr/local/lib/libopal.so.3.10-beta2(_ZN16OpalJitterBuffer8ReadDataER13RTP_D
ataFrameRK13PTimeInterval+0x1652)[0x7fe966a7486c]

/usr/local/lib/libopal.so.3.10-beta2(_ZN22OpalJitterBufferThread8ReadDataER1
3RTP_DataFrame+0x48)[0x7fe966a74d4c]

/usr/local/lib/libopal.so.3.10-beta2(_ZN11RTP_Session16ReadBufferedDataER13R
TP_DataFrame+0x7f)[0x7fe966a61747]

/usr/local/lib/libopal.so.3.10-beta2(_ZN18OpalRTPMediaStream10ReadPacketER13
RTP_DataFrame+0x98)[0x7fe966a413aa]

/usr/local/lib/libopal.so.3.10-beta2(_ZN14OpalMediaPatch4MainEv+0x20c)[0x7fe
966a491c2]

/usr/local/lib/libopal.so.3.10-beta2(_ZN14OpalMediaPatch6Thread4MainEv+0x33)
[0x7fe966a4be07]

/usr/local/lib/libpt.so.2.10-beta2(_ZN7PThread14PX_ThreadStartEPv+0x147)[0x7
fe9660e0e83]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b40)[0x7fe965188b40]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe96547336d] 

 

Since "codectest" works without errors, I suppose it have something to do
with memory distribution of current program.

It's looks like wrong memory being overwritten. So I tried to use valgrind
and find it reported errors like this:

 

==14162== Thread 12:

==14162== Invalid write of size 8

==14162==    at 0x4028F74: memcpy (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)

==14162==    by 0x5F93D8A: std::basic_streambuf<char, std::char_traits<char>
>::xsgetn(char*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)

==14162==    by 0x5F6A40D: std::basic_filebuf<char, std::char_traits<char>
>::xsgetn(char*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)

==14162==    by 0x5F74882: std::istream::read(char*, long) (in
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16)

==14162==    by 0xF8AE629: H264EncCtx::readStream(char*, unsigned int)
(h264pipe_unix.cxx:235)

==14162==    by 0xF8ADDA7: H264EncCtx::call(unsigned int, unsigned char
const*, unsigned int&, unsigned char*, unsigned int&, unsigned int&,
unsigned int&, int&) (h264pipe_unix.cxx:177)

==14162==    by 0xF8A4E1D: MyEncoder::Transcode(void const*, unsigned int&,
void*, unsigned int&, unsigned int&) (h264-x264.cxx:912)

==14162==    by 0xF8A6631:
PluginCodec<x264>::Transcode(PluginCodec_Definition const*, void*, void
const*, unsigned int*, void*, unsigned int*, unsigned int*)
(opalplugin.hpp:447)

==14162==    by 0x55306CF: OpalPluginTranscoder::Transcode(void const*,
unsigned int*, void*, unsigned int*, unsigned int*) const
(opalpluginmgr.h:228)

==14162==    by 0x5526ACD:
OpalPluginVideoTranscoder::EncodeFrames(RTP_DataFrame const&,
PList<RTP_DataFrame>&) (opalpluginmgr.cxx:929)

==14162==    by 0x5526872:
OpalPluginVideoTranscoder::ConvertFrames(RTP_DataFrame const&,
PList<RTP_DataFrame>&) (opalpluginmgr.cxx:893)

==14162==    by 0x509FE51: OpalMediaPatch::Sink::WriteFrame(RTP_DataFrame&)
(patch.cxx:892)

==14162==  Address 0x85aa708 is 1,560 bytes inside a block of size 1,564
alloc'd

==14162==    at 0x4027147: operator new(unsigned long) (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)

==14162==    by 0x5C3DE0F: PAbstractArray::PAbstractArray(int, int)
(contain.cxx:182)

==14162==    by 0x5070B08: PBaseArray<unsigned char>::PBaseArray(int)
(array.h:287)

==14162==    by 0x50C1C0B: PBYTEArray::PBYTEArray(int) (array.h:691)

==14162==    by 0x50B2D82: RTP_DataFrame::RTP_DataFrame(int, int)
(rtp.cxx:75)

==14162==    by 0x55269F2:
OpalPluginVideoTranscoder::EncodeFrames(RTP_DataFrame const&,
PList<RTP_DataFrame>&) (opalpluginmgr.cxx:920)

==14162==    by 0x5526872:
OpalPluginVideoTranscoder::ConvertFrames(RTP_DataFrame const&,
PList<RTP_DataFrame>&) (opalpluginmgr.cxx:893)

==14162==    by 0x509FE51: OpalMediaPatch::Sink::WriteFrame(RTP_DataFrame&)
(patch.cxx:892)

==14162==    by 0x509EC6D: OpalMediaPatch::DispatchFrame(RTP_DataFrame&)
(patch.cxx:709)

==14162==    by 0x509E244: OpalMediaPatch::Main() (patch.cxx:593)

==14162==    by 0x50A0E06: OpalMediaPatch::Thread::Main() (patch.h:309)

==14162==    by 0x5C02E82: PThread::PX_ThreadStart(void*) (tlibthrd.cxx:508)

 

I also try introspection approach and find that H264EncCtx::readStream
called with 1803 bytes requested before detected error and RTP_DataFrame was
initialized with RTP_DataFrame(540,1564). 

So it's looks like RTP_DataFrame buffer overridden by data returned from
x264 encoder thread. 

My question is: in which direction must I look for this problem from now? Is
it buffer too small or returned amount of data to big?