#163 Memory issues with tiling manager

[Anonymous]

Originally created by: kanto... (code.google.com)@gmail.com

What steps will reproduce the problem?
1. Start Flightgear using custom scenery such as France
2. Fly for 3 - 5 minutes with UFO
3. Check memory usage

What is the expected output? What do you see instead?
Memory usage increases all the time as you keep flying. This is especially visible with large polycount meshes and large visibility distance.
I expect to see older tiles deleted from memory and memory usage to vary only due to mesh complexity and object count. Tested without dynamic objects and trees.
In tilemgr.cxx FGTileMgr::sched_tile this line:
if ( tile_cache.insert_tile( e ) ) {
Always returns true, even if cache size is larger than maximum size as evidenced by:
TileCache::insert_tile in Simgear TileCache.cxx which returns true always.
This has the potential to leave a dangling object of type TileEntry.
Attached potential patch for tilemgr.cxx but ignore the debugging output lines.
This does not solve the main problem of the memory creep-up which leads me to believe the issue is inside either SceneryPager or osg::DatabasePager

What version of the product are you using? On what operating system and
what graphics card?

Latest GIT on Debian stable and Nvidia card.

Please provide any additional information below.
One aditional issue is that inside TileCache::get_oldest_tile() if the tileEntry has not loaded yet the return value is -1, leading to additional bogus inserts of TileEntries in code submitted.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

I don't think the patch is a solution. "tile_cache.get_max_cache_size()" is not a hard limit for the cache size - it's just a "hint" calculated depending on visibility. It's normal that there may be more elements in the cache than "max" size, this does _not_ cause bogus insertions/corruption. When visibility reduces there will always be more tiles in the cache then given by max size (at least for a while). This is ok...
Only when a new tile is to be inserted does "FGTileMgr::sched_tile" always try to reduce the cache to the hinted maximum (" while (tile_cache.get_size() >= tile_cache.get_max_cache_size() ) {..}" loop). Indeed, it could happen that this loop doesn't successfully reduce the cache, when all the tiles are either locked (= required for the current view) or not fully loaded yet. But in this case the new tile still has to be inserted. There is no other option, since FG won't try to load the tile again. Not inserting causes empty scenery patches, so adding "if ((int)tile_cache.get_size() < tile_cache.get_max_cache_size())" is not good.
If memory consumption really increases (indefinitely) then the root cause has to be elsewhere - e.g. maybe there is tiles which never finish loading, or maybe deleting tiles does not free all associated memory?

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

@kantooon: Can you give some more details on your setup? I checked but don't see any issues with the tile manager in a "normal" setup. Also, I don't see the memory consumption increase over time (remains about constant after 2 minutes).

However, I guess your problem might be related to your issue #162. Which visibility exactly are you specifying on command-line?
In a normal setup (e.g. with real weather etc), the visibility requires the tile cache to hold roughly between 32-96 tiles. A number even old PCs can handle.

In fact, I can reproduce a problem when I tune visibility to something insane, e.g. 100miles or more. This increases the tile cache to more than 1000 tiles.
My system then takes much, much longer loading all the tiles required for any location than it takes the UFO to move to a new location. This means only few tiles can be kicked from the cache since most of them are not loaded yet. When the UFO keeps moving fast the system keeps slowing down more and more, as the number of tiles busy loading increases all the time - and each new tile takes longer and longer because so many others are already loading...

So, using a higher visibility than the system (CPU/hard disk) can handle, eventually causes FlightGear to lock up (and eat up all memory). Not nice. The easiest work-around is to keep visibility at a level the system can handle. On my system the limit is at about 50miles...

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

brehmt: thanks for responding. My setup is a dual core AMD@3000Mhz with 2 GB RAM. Video Nvidia 9800GT. Visibility via argument between 20 miles and 50. This issue is most visible when loading custom scenery.
At 20 miles visibility, VmSize at start is 1.3 GB. After 5 minutes UFO flight increases by 200 MB.
At 35 miles visibility VmSize at start is 1.5 GB. After 5 minutes UFO in any direction, VmSize goes beyond 2 GB and enters swap with predictable result.
At 50 miles things don't take that long. Left for some time swap size is not enough and the program dies from memory starving.
In order to be able to make my tests I have decreased cache size by half (don't care about the tower issue for now).
Now, I'm sorry if I come over as total bloody idiot, since I am not a C++ programmer.
But this code:
// insert the tile into the cache
        if ( tile_cache.insert_tile( e ) ) {
        e->set_inner_ring( is_inner_ring );
        e->set_cache_lock( is_cache_locked );
        // update_queues will generate load request
        // Attach to scene graph
        e->addToSceneGraph(globals->get_scenery()->get_terrain_branch());
        } else {
        // insert failed (cache full with no available entries to
        // delete.)  Try again later
        delete e;
        }

does not seem to do what it should, since tile_cache.insert_tile() always returns true no matter if the cache size is too big. Related to that, the while loop that does the deleting part sometimes returns -1 as I could observe by a couple of cout lines. This means no entry is deleted, but one entry is added. Tile cache size can double the maximum size before any tile can be deleted. This can be observed in log2.txt attached. Please ignore "zaxis is" spam. With my patch, no tile is inserted before one is deleted (if the cache is at max size). This, however, does not solve the main issue. Memory keeps creeping up at the rates I have mentioned above.
If the maximum cache size is left at double the required value, 300 tiles sometimes get loaded at the same time. I believe the tower issue should be solved in other way than by doubling the cache size. But this is not important. The problem is memory increase despite leaving areas of complex mesh (mountains) to areas of low complexity.
Please try with custom France scenery, I believe you will see the issue faster.
This makes it impossible to fly for a long period over custom terrain.

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

As a side note: given that my patch does not solve the memory issue, and the fact that I have tried disabling all dynamic models and trees, could this be a problem with DatabasePager? If one flies over the same location, the memory does not creep up, which led me to investigate the scenery manager. However I have yet to read the implementation of DatabasePager. Somehow to me it looks like the osg::Lod node associated with every tile does not get paged out of memory once it is no longer attached to the scenegraph. I should mention that I have tried the multithreading option both disabled and enabled.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Ok, thanks kantooon. Yes, you're right that the "if ( tile_cache.insert_tile( e ) )" doesn't have a meaning right now, since "insert_tile" never reports a problem. But this is not the root cause for your issue. New tiles _have_ to be inserted, otherwise this results in other problems (the "try again later" comment is misleading - it won't...). We could even remove the entire condition for now. Only if someone added a memory check in "insert_tile" would this have a meaning. But if this really happened, it's too late anyway (no more RAM/swap space - FG is probably already doomed...).
The root problem is that either tiles cannot be kicked from the cache, or memory isn't freed properly when tiles are kicked.

I'll try to reproduce this. Can you exactly state where you start your flight (airport?), how fast are you going (max UFO speed?) and which scenery are you using (terrasync? additional scenery - which?).

And, oh, you can't just reduce the tile cache size by half. You can only reduce visibility. The cache size has to be about twice the size required for the current visibility, since tiles for the current view are locked - and the cache must also load tiles for a new location... If you reduced the cache size by half, it's obvious that tiles were sometimes only kicked when the cache was filled to twice the "maximum"... Again, the "max_cache_size" is not a hard limit. It's just a hint to determine how many tiles should be kicked from the cache, if possible.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

@kantooon: since you know to use patches, could you apply the attached diff (simgear), reproduce your problem and post your output?
It adds statistics to check if there was a problem with the cache itself - and if, why tiles are blocked...

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

Ok, I can see the point about the cache size. I only increased it by 2 in order to spare some memory to complete the tests.
UFO speed between half and maximum.
Airport LFLS using custom France scenery: http://wiki.flightgear.org/index.php/Custom_France_Scenery
Arguments:
--log-level=4
--control=joystick
--enable-random-objects
--disable-hud-3d
--enable-horizon-effect
--enable-enhanced-lighting
--enable-distance-attenuation
--turbulence=0.16
--enable-clouds3d
--enable-fullscreen
--prop:/sim/frame-rate-throttle-hz=31
--prop:/sim/menubar/autovisibility/enabled=1
--fog-nicest
--geometry=1920x1024
--visibility-miles=20
--bpp=32
--fov=90
--texture-filtering=8
--timeofday=noon
--disable-real-weather-fetch

Heading towards East, lots of mountains, memory usage increases, heading towards West then, memory keeps increasing.

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

Will do and report back.

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

I have reverted all my patches and applied yours. Could not obtain any output.
Starting from LFLS with visibility 30 miles, initial memory size is 1 GB. Moving just 2-3 km west of the airport and stopping, memory size increases by 300 MB. Moving East 2-3 km, memory increases by another 300 MB. Moving south memory continues to increase until out of memory. I would expect that after a while the older tiles would be deleted right?

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Hmm. No output means the tile manager is not the problem. And if you only moved 2-3km it can't be anyway. It can't be true that the French scenery maybe is just so detailed/incredibly huge that this was normal? I'll by testing the French scenery...

I already found another problem, however it cannot cause a huge leak. Flying over empty scenery tiles (tiles that are not installed on the system and cannot be loaded) causes their empty "TileEntry"s to sit in the tile cache forever, since their "is_loaded" never returns true (right, what's not installed cannot be loaded - but it's handle still needs to be kicked from the cache sometime...). Results in a tiny memory leak - and in the tile cache growing all the time. Well, I guess not too many people enjoy flying across empty scenery patches. Still needs to be fixed. I'll be asking Tim about this one...

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

Here's my take on it: even with the cache reduced to half, memory still increases. I'd expect it to stop at some point, or to decrease when going over the ocean right?
I think there is something fishy there, but can't put my finger on it. I still have to find out why sometimes the index returned was -1 despite the fact that the cache was twice it's maximum size. Could have been related to my issue #162.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

I've tested the French custom scenery now. Beautiful. But *very* detailed.

With 20m visibility I see the memory start at 1GB and then increase to a maximum of 1.7GB. Seems I can fly forever without additional increase then. Requirements for the standard scenery (terrasync) are a lot lower.

I'm not quite sure how this works, but I guess the initial increase is a result of generic elements being loaded (e.g. houses, textures etc). I guess these are loaded as soon as any tile needs them. And if they were never dropped from RAM, memory increases until all generic elements are loaded.
@kantooon: I have a similar setup like yours - but a bit more RAM. I'm afraid your 2GB might just not be enough to use this detailed scenery. How about adding more? ;-)

With the detailed French scenery, 50 miles is way beyond my PCs capabilities. Too many/too detailed tiles are loaded then. Only if I move *very* slowly it kind of works. Maximum RAM needed then is about 2.5GB. But if I don't move slow enough, FG fails to keep up with loading, which means the number of requested tiles is ever increasing, slowing the system down more and more, increasing RAM size more and more - until it locks up...

=> So, the way the system is built right now, you just mustn't use higher visibility levels than your system can handle (loading speed/RAM size). Otherwise it starts thrashing... Apart from that, I'm not seeing an unlimited memory increase.

Cache strategy could still be improved. FG behaves much better in overload situations when the "e->is_loaded()" check in "TileCache::get_oldest_tile" is dropped (attached patch). Even if a tile is not yet (fully) loaded, it could still be dropped when it's too old and no longer needed. Fixes the tile cache problem when flying over uninstalled scenery areas. And when the system is overloaded, it's better to spend time loading tiles needed now, than to finish loading outdated ones... But I'm not sure if it's safe to drop the check. Need to ask Tim...

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

brehmt: I have dropped the check before. I agree that France requires too much memory. However. In my attempt to fix that on my machine, I have added the max_cache_size check in tilemgr.cxx, I have removed the doubling of the max_cache_size in schedule_needed() and I have patched TileCache.cxx to return the tile index even if the tile has not been loaded yet.
With 35 miles visibility, cache size is about 80 tiles, and new tiles are only inserted after deleting the oldest one. That can be seen in my first log.txt.
Even in that situation, with only 80 tiles at one time, memory keeps increasing. When I set the /environment/visibility-m correctly at start (see #162), cache gets filled.
After that, when I begin to move, tiles get dropped and loaded all time and the number of tiles in the cache is constant. But the memory size increases constantly.
Why is that happening, I can't tell.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

There are several caches involved (tiling manager in FG, another cache handled by the OSG library). So memory may even somewhat increase when the tile manager has a fixed amount of tiles) - since the OSG cache may still keep some data (which is normal).
Also, memory needs of tiles may be very different - ocean tiles need almost no memory, many of the French custom scenery tiles need a lot. So moving between tiles may increase memory. And of course, the caching behaviour of the specific OSG version used could also influence memory needs. But at least on Linux/OSG 2.8.3/current GIT I'm not seeing a (huge) memory leak when flying in the French scenery.

Status: Testing

[Anonymous]

Originally posted by: kanto... (code.google.com)@gmail.com

I'm not up-to-date with current GIT, but it may well be that memory requirements are greater then what my machine has to offer. If nobody else has reported this issue, this behaviour may be normal.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Several issues were fixed with the TileManager updates. Haven't seen any scenery related memory issues since then. And no new reports. => fixed

Status: Fixed

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

(No comment was entered for this change.)

Status: Temp

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

(No comment was entered for this change.)

Status: Fixed

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

(No comment was entered for this change.)

Labels: Milestone-2.4.0