RCDcap failed after few hours

[Julien]

Hello,

RCDcap crash after few hours.
The command is :
rcdcap -i eth0 --erspan -d -b 400MB --tap-persist -t ERSPAN --expression "host IP1 or host IP2"

The rcdcap log :

[root@VTAP-RCV-03 ~]# cat /var/log/rcdcap
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:156: RCDCap::PacketInfo* RCDCap::CommonBuffer::next(RCDCap::PacketInfo*) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.
worker threads: 6 on 6 CPUs
terminate called after throwing an instance of 'RCDCap::Exception'
  what():  /tmp/RCDCap-0.7.99-Source/src/sink.cc:307: Invalid length; the complete packet must be captured in order to output it to a TAP device.
Original length: 3337402711; the length of the received packet: 4025604810.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:140: void RCDCap::CommonBuffer::popSequence(size_t): Assertion `bytes <= m_Count' failed.
worker threads: 6 on 6 CPUs
terminate called after throwing an instance of 'RCDCap::Exception'
  what():  /tmp/RCDCap-0.7.99-Source/src/sink.cc:307: Invalid length; the complete packet must be captured in order to output it to a TAP device.
Original length: 1818326560; the length of the received packet: 577073273.
worker threads: 6 on 6 CPUs
rcdcap: /tmp/RCDCap-0.7.99-Source/src/common-buffer.cc:140: void RCDCap::CommonBuffer::popSequence(size_t): Assertion `bytes <= m_Count' failed.
worker threads: 6 on 6 CPUs

The trafic load is about 8 MB/s and 17K pps.

Do you have an idea ?

Regards,
Julien

[Zero effect]

Are you sure that you don't have jumbo frames enabled? You may increase the packet size by using -s 1600 or -s 9100 if you have jumbo frames.

[Julien]

I tried both and I had this error :

rcdcap: /tmp/RCDCap-0.7.99-Source/src/sink.cc:275: void RCDCap::TAPDeviceSink::writeCompleted(RCDCap::PacketInfo*, size_t, size_t, const boost::system::error_code&, std::size_t): Assertionbytes_transferred == packet_info->getPCAPHeader().getCapturedLength()' failed.`

[Julien]

For your information, it works fine without the erspan decode!
It seems there is an issue into it.

[Zero effect]

Can you check the latest revision? I have added a very simple option to bypass the check and drop packets (-y or --ignore-incomplete)

[Julien]

Thx for your time.
I tried and sadly it does not work :

80610954 packets were captured by the application
80611383 packets were captured by the kernel
283 packets were dropped by the kernel
0 packets were dropped by the driver
0 packets were dropped due to buffer overflow
9783068 ms execution time
80610954 packets were processed
worker threads: 6 on 6 CPUs
rcdcap: /tmp/source/src/common-buffer.cc:156: RCDCap::PacketInfo RCDCap::CommonBuffer::next(RCDCap::PacketInfo) const: Assertion `reinterpret_cast<char*>(packet_info) + packet_info->getAllocatedSize() <= m_Data + m_Capacity' failed.

[Zero effect]

Alright, can you try the latest modification? My knowledge about this pipeline was getting a little bit rusty. It would be much faster to fix it if you can provide a repro dump of a stream of packets.