Menu
▾
▴
1 Attachments
#2831 Crash when generating random vegetation. heap-use-after-free
New
High
2024-12-01
2023-10-03
No
I get this crash every time when I enable random vegetation.
The density parameter only changes when it's going to crash and not if.
Built with GCC 12, Crash happens with OSG 3.6 and master. And on FG 2020.3 and Next.
AddressSanitizer/Backtrace
909.14 [INFO]:terrain Loading tile 3154393.stg
909.14 [INFO]:terrain Loading stg file Path "/home/reglnx/FlightGear/FlightGear-2020.3/Downloads/TerraSync/Terrain/e010n40/e012n45/3154393.stg"
910.55 [INFO]:terrain Loading tile 3154395.stg
910.55 [INFO]:terrain Generating ocean tile: e010n40/e012n45/3154395
910.55 [INFO]:terrain Loading tile 3170793.stg
910.56 [INFO]:terrain Loading stg file Path "/home/reglnx/FlightGear/FlightGear-2020.3/Downloads/TerraSync/Terrain/e010n40/e013n45/3170793.stg"
917.45 [INFO]:terrain Loading tile 3138019.stg
917.45 [INFO]:terrain Loading stg file Path "/home/reglnx/FlightGear/FlightGear-2020.3/Downloads/TerraSync/Terrain/e010n40/e011n45/3138019.stg"
918.35 [INFO]:terrain Got ground pick at lon = 12.6529deg, lat = 45.7907deg, elev = 3.66493m
919.27 [INFO]:terrain Loading tile 3154392.stg
919.27 [INFO]:terrain Loading stg file Path "/home/reglnx/FlightGear/FlightGear-2020.3/Downloads/TerraSync/Terrain/e010n40/e012n45/3154392.stg"
939.38 [INFO]:environment FGEnvironmentMgr::updateClosestAirport: selected:LIDW
949.22 [INFO]:environment FGEnvironmentMgr::updateClosestAirport: selected:LIDK
980.51 [INFO]:terrain FGStgTerrain::reinit - reinit tilemgr
980.51 [INFO]:terrain Initializing Tile Manager subsystem.
980.51 [INFO]:terrain Initializing the tile cache.
980.51 [INFO]:terrain max cache size = 640
980.51 [INFO]:terrain current cache size = 300
=================================================================
==848082==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0193405ec at pc 0x7f7ee234d4fc bp 0x7f7ea4b111e0 sp 0x7f7ea4b111d8
WRITE of size 4 at 0x60e0193405ec thread T35
#0 0x7f7ee234d4fb in osg::BufferObject::removeBufferData(unsigned int) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2324fb)
#1 0x7f7ee234e5eb in osg::BufferData::setBufferObject(osg::BufferObject*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2335eb)
#2 0x7f7ee234e7eb in osg::BufferData::~BufferData() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2337eb)
#3 0x55df46be23d3 in osg::TemplateArray<osg::Vec3f, (osg::Array::Type)28, 3, 5126>::~TemplateArray() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0xfeb3d3)
#4 0x7f7ee2680071 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565071)
#5 0x7f7ee2680bc0 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565bc0)
#6 0x7f7ee274cd61 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d61)
#7 0x7f7ee5684f4a in simgear::EffectGeode::~EffectGeode() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0x681f4a)
#8 0x7f7ee274cd81 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d81)
#9 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#10 0x7f7ee274cd61 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d61)
#11 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#12 0x7f7ee274cd41 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d41)
#13 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#14 0x7f7ee274cd21 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d21)
#15 0x7f7ee2938100 in osg::MatrixTransform::~MatrixTransform() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x81d100)
#16 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#17 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#18 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#19 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#20 0x7f7ee46ed719 in osgDB::DatabasePager::DatabaseThread::run() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgDB.so.162+0x2cb719)
#21 0x7f7ee21144bd in OpenThreads::ThreadPrivateActions::StartThread(void*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libOpenThreads.so.21+0xa4bd)
#22 0x7f7edf26f043 in start_thread nptl/pthread_create.c:442
#23 0x7f7edf2ef5fb in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
0x60e0193405ec is located 76 bytes inside of 152-byte region [0x60e0193405a0,0x60e019340638)
freed by thread T0 here:
#0 0x7f7ee63b9f08 in operator delete(void*) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:152
#1 0x7f7ee2680071 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565071)
#2 0x7f7ee2680bc0 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565bc0)
#3 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#4 0x7f7ee5684f4a in simgear::EffectGeode::~EffectGeode() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0x681f4a)
#5 0x7f7ee274cdc5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631dc5)
#6 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#7 0x7f7ee274cddc in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631ddc)
#8 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#9 0x7f7ee274cd61 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d61)
#10 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#11 0x7f7ee274cd81 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d81)
#12 0x7f7ee2938100 in osg::MatrixTransform::~MatrixTransform() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x81d100)
#13 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#14 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#15 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#16 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#17 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#18 0x7f7ee29b9d50 in osg::PagedLOD::~PagedLOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x89ed50)
#19 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#20 0x7f7ee2938100 in osg::MatrixTransform::~MatrixTransform() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x81d100)
#21 0x7f7ee274cdc5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631dc5)
#22 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#23 0x7f7ee274cdc5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631dc5)
#24 0x7f7ee29b9d50 in osg::PagedLOD::~PagedLOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x89ed50)
#25 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#26 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#27 0x55df47e0b291 in TileEntry::~TileEntry() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x2214291)
#28 0x55df47e012c6 in TileCache::entry_free(long) (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x220a2c6)
#29 0x55df47e02347 in TileCache::clear_cache() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x220b347)
previously allocated by thread T35 here:
#0 0x7f7ee63b94c8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
#1 0x7f7ee5c7d90c in simgear::createTreeGeometry(float, float, int) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0xc7a90c)
#2 0x7f7ee5c848fe in simgear::createTreeGeode(float, float, int) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0xc818fe)
#3 0x7f7ee5c8b2e7 in simgear::createForest(std::__cxx11::list<simgear::TreeBin*, std::allocator<simgear::TreeBin*> >&, osg::Matrixd const&, simgear::SGReaderWriterOptions const*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0xc882e7)
#4 0x7f7ee5d81378 in SGTileDetailsCallback::readNode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, osgDB::Options const*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0xd7e378)
#5 0x7f7ee46eeff8 in osgDB::DatabasePager::DatabaseThread::run() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgDB.so.162+0x2ccff8)
#6 0x7f7ee21144bd in OpenThreads::ThreadPrivateActions::StartThread(void*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libOpenThreads.so.21+0xa4bd)
#7 0x7f7edf26f043 in start_thread nptl/pthread_create.c:442
Thread T35 created by T0 here:
#0 0x7f7ee6349726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
#1 0x7f7ee2113825 in OpenThreads::Thread::start() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libOpenThreads.so.21+0x9825)
#2 0x7f7ee46eb2f6 in osgDB::DatabasePager::requestNodeFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<osg::Node*, std::allocator<osg::Node*> >&, float, osg::FrameStamp const*, osg::ref_ptr<osg::Referenced>&, osg::Referenced const*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgDB.so.162+0x2c92f6)
#3 0x55df47dbbaff in flightgear::SceneryPager::PagerRequest::doRequest(flightgear::SceneryPager*) (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x21c4aff)
#4 0x55df47dbee0a in flightgear::SceneryPager::signalEndFrame() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x21c7e0a)
#5 0x7f7ee355472f in osgViewer::ViewerBase::renderingTraversals() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgViewer.so.162+0x32072f)
#6 0x55df48a65571 in fgOSMainLoop() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x2e6e571)
#7 0x55df48e2dbf1 in fgMainInit(int, char**) (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x3236bf1)
#8 0x55df4647b46a in main (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0x88446a)
#9 0x7f7edf20d1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
SUMMARY: AddressSanitizer: heap-use-after-free (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2324fb) in osg::BufferObject::removeBufferData(unsigned int)
Shadow bytes around the buggy address:
0x0c1c83260060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c1c83260070: 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 00 00
0x0c1c83260080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c1c83260090: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x0c1c832600a0: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
=>0x0c1c832600b0: fa fa fa fa fd fd fd fd fd fd fd fd fd[fd]fd fd
0x0c1c832600c0: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
0x0c1c832600d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c1c832600e0: 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 00 00
0x0c1c832600f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c1c83260100: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
AddressSanitizerAddressSanitizer:DEADLYSIGNAL
:DEADLYSIGNAL
=================================================================
==848082==ERROR: AddressSanitizer: SEGV on unknown address 0x60e01df1a7ac (pc 0x7f7ee234c0fc bp 0x7f7ea4b114b0 sp 0x7f7ea4b111f0 T35)
==848082==The signal is caused by a WRITE memory access.
#0 0x7f7ee234c0fc in osg::BufferObject::removeBufferData(unsigned int) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2310fc)
#1 0x7f7ee234e5eb in osg::BufferData::setBufferObject(osg::BufferObject*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2335eb)
#2 0x7f7ee234e7eb in osg::BufferData::~BufferData() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2337eb)
#3 0x55df46be23d3 in osg::TemplateArray<osg::Vec3f, (osg::Array::Type)28, 3, 5126>::~TemplateArray() (/home/reglnx/FlightGear/FlightGear-2020.3/bin/fgfs+0xfeb3d3)
#4 0x7f7ee2680051 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565051)
#5 0x7f7ee2680bc0 in osg::Geometry::~Geometry() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x565bc0)
#6 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#7 0x7f7ee5684f4a in simgear::EffectGeode::~EffectGeode() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libSimGearScene.so.2020.3.18+0x681f4a)
#8 0x7f7ee274cd81 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d81)
#9 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#10 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#11 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#12 0x7f7ee274cd41 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d41)
#13 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#14 0x7f7ee274cd41 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631d41)
#15 0x7f7ee2938100 in osg::MatrixTransform::~MatrixTransform() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x81d100)
#16 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#17 0x7f7ee4a3e4eb in osg::LOD::~LOD() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgSim.so.162+0xc34eb)
#18 0x7f7ee274cda5 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x631da5)
#19 0x7f7ee274d170 in osg::Group::~Group() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x632170)
#20 0x7f7ee46ed719 in osgDB::DatabasePager::DatabaseThread::run() (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosgDB.so.162+0x2cb719)
#21 0x7f7ee21144bd in OpenThreads::ThreadPrivateActions::StartThread(void*) (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libOpenThreads.so.21+0xa4bd)
#22 0x7f7edf26f043 in start_thread nptl/pthread_create.c:442
#23 0x7f7edf2ef5fb in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/home/reglnx/FlightGear/FlightGear-2020.3/lib/libosg.so.162+0x2310fc) in osg::BufferObject::removeBufferData(unsigned int)
==848082==ABORTING
This isn't specific to random vegetation: it's a general problem in removeBufferData. Right now we never found a way to tirgger it reliably, so if you have a way to make it happen reliably that would be extremely useful since it also happens with WS3!
We also see this crash in the wild from users, always when unloading tiles/scenery or quitting the sim.
Ahhh! It's the DBPager and the tile-caching fighting over who gets to delete something. That is veyr interesting, let me go check the other crash reports from users and see if the same pattern exists.
--edited--
Last edit: Megaf 2023-10-04
Set:
Basic Weather, Fair Weather
And set random vegetation to max.
Take off from LIPA. You should get a crash every time, often as soon as the sim starts loading the scenery.
Last edit: Megaf 2023-10-04
Doing a code read, I wonder if we shouldn't simply leave the DatabasePager to handle the deletion of everything except for the OceanTiles.
E.g. change TileEntry::removeFromSceneGraph() so that it only calls removeChild() for OceanTiles?
@stuartbuchanan yeah I don't know why we have this scheme of different people dpoing the deletion :/