FlightGear - Flight Simulator

Am 12.07.2011 10:18, schrieb thorsten.i.renk@...:
>
> Well, there's also a reason - the different design philosophy - and at
> some point you may want to consider that before you merge.
Rest assured, there won't be any merge of the weather system without you ;-)
>
> If you compare a system that tries to understand atmospheric conditions
> and terrain interaction with one that draws what you specify, then there
> are pros and cons to each:
I have to postpone this discussion to after the release which completely 
consumes all my time I am currently able to dedicate to fg.

>>> Do you see this as a problem with the 3D clouds generated by the Local
>>> [..]
>> It's 3d clouds in general but the local weather has the biggest impact
>> [..]
> In a single monitor setup, it's just not true that Global Weather is
> [..]
The issue is with 3d-clouds. It's not a question of global/local 
weather. Because local weather relies on 3d clouds, it appears to be 
much slower than global weather with 3d clouds disabled. My conclusion 
is, that our current 3d cloud implementation does not scale very well 
with screen size.
>
> If the problem gets worse when a single cloud is in view, it could be down
> to texture resolution - a single cloudlet texture used by Stuart is
> probably 120x120 whereas some of my large Cu cloud lets are ~1024x512 -
> that should give you a different performance footprint... Some people had
> issues with GPU memory, in which case dds textures helped (didn't help for
> me).
Our four GTX460 have 2GB GPU memory, each. That's probably not the 
bottleneck.

I hope I have some more time for discussing this in a just a few weeks.
Thanks,
  Torsten

> -----Original Message-----
> From: thorsten.i.renk@... [mailto:thorsten.i.renk@...]
> Sent: 12 July 2011 09:18
> To: FlightGear developers discussions
> Subject: Re: [Flightgear-devel] Future Weather System
> 
> > What I'd really love to see in the mid-to-long-term range is some kind
> > of unified weather system. It does not really make sense for an average
> > user to have two systems to choose from.
> 
> Well, there's also a reason - the different design philosophy - and at
> some point you may want to consider that before you merge.
> 
> If you compare a system that tries to understand atmospheric conditions
> and terrain interaction with one that draws what you specify, then there
> are pros and cons to each:
> 
> * currently Global Weather guarantees (I think) that winds at a station
> position are exactly as reported in the METAR string. Local Weather
> computes the boundary layer wind slowdown locally dependent on terrain and
> can't guarantee that the winds will work out - by the time a METAR is
> received, the terrain at the station usually isn't even loaded and can't
> be probed, so the system has to make a guess what the boundary effect at
> the station location will be, and the wind at destination is only good up
> to that guess (that could in principle be addressed by correcting the
> guess once the terrain is loaded - it's just a nasty question of timing
> and subtly changing interpolation points...).
> 
> * same with cloud cover - if a system computes cloud placement dependent
> on terrain type and altitude, you can't guarantee that the end result will
> really be a 4/8 - it could come out 3/8 or 5/8 - again, it depends on
> guessing a factor before doing the placement
> 
> * on the other hand, if you place and drift clouds without terrain
> interaction, they'll just go through a mountain and emerge on the other
> side - isn't quite right either. If you compute winds without local
> boundary layer effect, the boundary layer will be as effective on a
> mountaintop as down in the valley - isn't correct either.
> 
> Maybe I am lost in seemingly irrelevant detail - but I think there's a
> good reason to have two systems dependent on what you need - either
> accuracy with respect to weather reports, or some physics model of the
> atmosphere interacting with the terrain.
> 
> (There is of course a proper solution, which is a proper physics model of
> atmosphere/terrain interaction - it's just computationally too heavy...).
> 
> >> Do you see this as a problem with the 3D clouds generated by the Local
> >> Weather system specifically, or 3D clouds in general?
> > It's 3d clouds in general but the local weather has the biggest impact
> > on frame rate. I think (better: guess) it's because Local Weather draws
> > more cloudlets. It's getting worse btw. the closer one gets to a cloud
> > and the more space on the monitor is occupied by a cloud.
> 
> I'm not sure about what's different in multiple monitors, but:
> 
> http://www.flightgear.org/forums/viewtopic.php?f=5&t=7358&start=345#p12442
> 0
> 
> (visual comparison with closed layers in METAR mode - Local Weather gets
> almost 25% better framerate)
> 
> In a single monitor setup, it's just not true that Global Weather is
> generically faster. In equal Cumulus conditions, I observe about the same
> performance hit (when slecting the same visual range to display clouds -
> when I compare 55 km visibility with 20 km, naturally 20 km are better).
> In overcast layers, I observe that Local Weather is sigificantly better -
> sometimes twice as fast.
> 
> I think cloudlets go the other way round - Stuart uses many (20-30 per
> cloud) small cloudlets, whereas I use few large ones (typically 4-8) with
> asymmetric rescaling to get the layering right.
> 
> If the problem gets worse when a single cloud is in view, it could be down
> to texture resolution - a single cloudlet texture used by Stuart is
> probably 120x120 whereas some of my large Cu cloud lets are ~1024x512 -
> that should give you a different performance footprint... Some people had
> issues with GPU memory, in which case dds textures helped (didn't help for
> me).
> 
> Apart from that, I can't really guess what makes it worse with more
> screens.
> 
> So - future plans:
> 
> Stuart has written a very nice interface from Nasal, which I plan to use.
> Since that requires newly arranged texture sheets, I will ask some people
> who have more experience with graphics software to help me re-extract
> textures from my cloud photographs - at that stage, we can also
> systematically test with dds vs. rgb and optimal resolution.
> 
> My current understanding is (I haven't tested too much) that Stuart's
> interface doesn't address issues which are related to number of cloudlets
> or texture size - once a model is in the scene, these should be the only
> difference. I can of course use the same clouds and textures as currently
> in global weather, but then you run into the same performance and visual
> issues (i.e. cloudlets are too small to effectively form layers, no
> developed Cu or Cb clouds, ...). What the system will most likely do is
> 
> * improve loading times
> * provide a conceptually clean interface
> * move clouds in the wind almost for free
> 
> In principle, one could try to code terrain interaction into here as well
> - but as I said, moving clouds interacting with the terrain opens a can of
> worms...
> 
> So, that's in a nutshell the plan for the mid-future.
> 

Here, with a Core 2 Quad, 4Gb RAM, nVidia GTx285 with 2Gb VRAM there is a
huge difference in performance. At EGMH and using METAR, I get 75 fps with
Global Weather, but when I use Local Weather, using the same METAR, I get a
little over half that. The loss of frame rate wouldn't be so bad, but for
the stagger in the frame rate with very noticeable pauses. These pauses make
Local Weather a very much less pleasant experience to fly with than Global
Weather. I would even sacrifice a few more fps for the sake of smoothness.

And if we could loose the hard edges around some of the textures that would
help. Right now, the Cu look like a series of cardboard cutouts stacked one
behind the other, flicking on and off. OTOH, I love the appearance of Ci/Cc.


For me the main issue is not so much the framerate, as the way the framerate
is being delivered. Second to that, the flicking on/off detracts greatly
from the overall impression of what is potentially a very good system.

Vivian

On 12.07.2011 23:11, Vivian Meazza wrote:
> I would even sacrifice a few more fps for the sake of smoothness.
> For me the main issue is not so much the framerate, as the way the framerate
> is being delivered.

Indeed. Frame rate is misleading - the number only has a meaning if all 
frames were guaranteed to be evenly spaced (like on a TV/display). But 
that's not guaranteed for FG, so ignore this number.

In order to judge and compare visual quality, look at the worst-case 
delay in between two frames instead. If a single delay in between two 
frames is too high, the human eye immediately spots a "stutter". Doesn't 
help if all the other frames were produced at high rate. And if that 
stutter happens repeatedly (say every second), it's what limits visual 
quality.

You can enable a better property to compare performance using "View" => 
"Show worst-case frame delay". It shows the longest delay in between two 
frames within the last second of simulation (lower left corner). The 
lower the number, the better. In order to maintain an acceptable 25Hz 
simulation, the frame delay must never exceed 40ms.
Is anyone capable of running FlightGear with either global or local 
weather enabled with a frame spacing not exceeding 40ms?

cheers,
Thorsten

ThorstenB wrote

> -----Original Message-----
> From: ThorstenB [mailto:brehmt@...]
> Sent: 12 July 2011 22:40
> To: FlightGear developers discussions
> Subject: Re: [Flightgear-devel] Future Weather System
> 
> On 12.07.2011 23:11, Vivian Meazza wrote:
> > I would even sacrifice a few more fps for the sake of smoothness.
> > For me the main issue is not so much the framerate, as the way the
> framerate
> > is being delivered.
> 
> Indeed. Frame rate is misleading - the number only has a meaning if all
> frames were guaranteed to be evenly spaced (like on a TV/display). But
> that's not guaranteed for FG, so ignore this number.
> 
> In order to judge and compare visual quality, look at the worst-case
> delay in between two frames instead. If a single delay in between two
> frames is too high, the human eye immediately spots a "stutter". Doesn't
> help if all the other frames were produced at high rate. And if that
> stutter happens repeatedly (say every second), it's what limits visual
> quality.
> 
> You can enable a better property to compare performance using "View" =>
> "Show worst-case frame delay". It shows the longest delay in between two
> frames within the last second of simulation (lower left corner). The
> lower the number, the better. In order to maintain an acceptable 25Hz
> simulation, the frame delay must never exceed 40ms.
> Is anyone capable of running FlightGear with either global or local
> weather enabled with a frame spacing not exceeding 40ms?
> 

Local 60 - 120ms and anything in between ~ 40 fps indicated
Global 17 - 37ms ~ 75 fps indicated 

Both showed occasional excursions well outside these values. 

Using the Hurricane at EGMH and current METAR 

Vivian

Thorsten,

You mentioned earlier that a lot of the performance issues would disappear
if we could probe the terrain 100 times faster.  I've been thinking about
this for a while for ai traffic, skyop's moving map instrument, and weather.

I'm thinking of storing some resolution of altitude data in the tile itself,
making altitude queries very fast at the expense of a larger tile (in memory
and disk).

I'm just starting the work, but I would like any feedback on the idea.  It
may require some lod implementation if the base tile size gets too big.
I've been thinking about trying to get osg paged lod working with a new sg
bucket.

Obviously, this is a huge undertaking, but I hope to get a proof of concept
up and running in the next 6 months or so.
On Jul 12, 2011 6:24 PM, "Vivian Meazza" <vivian.meazza@...> wrote:
> ThorstenB wrote
>
>> -----Original Message-----
>> From: ThorstenB [mailto:brehmt@...]
>> Sent: 12 July 2011 22:40
>> To: FlightGear developers discussions
>> Subject: Re: [Flightgear-devel] Future Weather System
>>
>> On 12.07.2011 23:11, Vivian Meazza wrote:
>> > I would even sacrifice a few more fps for the sake of smoothness.
>> > For me the main issue is not so much the framerate, as the way the
>> framerate
>> > is being delivered.
>>
>> Indeed. Frame rate is misleading - the number only has a meaning if all
>> frames were guaranteed to be evenly spaced (like on a TV/display). But
>> that's not guaranteed for FG, so ignore this number.
>>
>> In order to judge and compare visual quality, look at the worst-case
>> delay in between two frames instead. If a single delay in between two
>> frames is too high, the human eye immediately spots a "stutter". Doesn't
>> help if all the other frames were produced at high rate. And if that
>> stutter happens repeatedly (say every second), it's what limits visual
>> quality.
>>
>> You can enable a better property to compare performance using "View" =>
>> "Show worst-case frame delay". It shows the longest delay in between two
>> frames within the last second of simulation (lower left corner). The
>> lower the number, the better. In order to maintain an acceptable 25Hz
>> simulation, the frame delay must never exceed 40ms.
>> Is anyone capable of running FlightGear with either global or local
>> weather enabled with a frame spacing not exceeding 40ms?
>>
>
> Local 60 - 120ms and anything in between ~ 40 fps indicated
> Global 17 - 37ms ~ 75 fps indicated
>
> Both showed occasional excursions well outside these values.
>
> Using the Hurricane at EGMH and current METAR
>
> Vivian
>
>
>
>
------------------------------------------------------------------------------
> AppSumo Presents a FREE Video for the SourceForge Community by Eric
> Ries, the creator of the Lean Startup Methodology on "Lean Startup
> Secrets Revealed." This video shows you how to validate your ideas,
> optimize your ideas and identify your business strategy.
> http://p.sf.net/sfu/appsumosfdev2dev
> _______________________________________________
> Flightgear-devel mailing list
> Flightgear-devel@...
> https://lists.sourceforge.net/lists/listinfo/flightgear-devel

> Here, with a Core 2 Quad, 4Gb RAM, nVidia GTx285 with 2Gb VRAM there is a
> huge difference in performance. At EGMH and using METAR, I get 75 fps
> with
> Global Weather, but when I use Local Weather, using the same METAR, I
> get a
> little over half that.

I hate to repeat myself, but what set of options (cloud view range,
dynamics on/off, dynamical convection on/off, thermals on/off are we
comparing here?

If you're running default settings, you're asking ~3 times the area cloud
coverage from Local Weather... naturally that is a bit slower.

If you have dynamical weather (= moving clouds) on, then indeed you're not
only losing plenty of framerate as compared to global but you're also
getting frequent garbage collection due to loads of Nasal work running per
frame, i.e. frame delays. Which is why we're moving cloud generation to
Stuart's system which moves clouds much more efficiently. We know that
already - in the mean time, if it's bothersome, switch dynamical weather
off.

You can (by maxing out all options) easily design a situation in which
Local Weather freezes your system to 6 frames. Rather than doing that, it
allows you to make a choice what is important for you and where you want
to use the resources.

I also don't get any flickering... so I can't really diagnose what is
going on, but basically you're looking at models in the scenery, so either
it's a shader problem (which'd be odd since it is basically the same
shader for all the clouds) or some deeper rendering issue - but nothing I
could fix. I have to rely on the Flightgear core to display a model
correctly if I ask it to place it into the scenery.

> And if we could loose the hard edges around some of the textures that
> would help.

Gimp does the trick. It's mainly down to someone doing it (either working
out an efficient way of cleaning the texture, or doing it pixel by pixel
manually) - it's not exceedingly high up on my to-do list.

Cheers,

* Thorsten

>> You can enable a better property to compare performance using "View" =>
>> "Show worst-case frame delay". It shows the longest delay in between two
>> frames within the last second of simulation (lower left corner). The
>> lower the number, the better. In order to maintain an acceptable 25Hz
>> simulation, the frame delay must never exceed 40ms.
>> Is anyone capable of running FlightGear with either global or local
>> weather enabled with a frame spacing not exceeding 40ms?

Quick test at KSEA with the F-16 and current METAR, cloud visibility range
20 km, dynamical weather off (as I said, that is indeed much slower):

Local 25-48 ms, 48 fps indicated
Global 25-50 ms 40 fps indicated


I hold to what I stated earlier...

* Thorsten

Hi,

On Wednesday, July 13, 2011 02:16:29 Peter Sadrozinski wrote:
> You mentioned earlier that a lot of the performance issues would disappear
> if we could probe the terrain 100 times faster.  I've been thinking about
> this for a while for ai traffic, skyop's moving map instrument, and
> weather.
> 
> I'm thinking of storing some resolution of altitude data in the tile
> itself, making altitude queries very fast at the expense of a larger tile
> (in memory and disk).
> 
> I'm just starting the work, but I would like any feedback on the idea.  It
> may require some lod implementation if the base tile size gets too big.
> I've been thinking about trying to get osg paged lod working with a new sg
> bucket.

While being able to do a croase ground query on such a kind of regular grid 
might be beneficial for the weather module. I would prefer the ai module just 
using the already available bounding volume tree that is used for the main 
aircrafts elevation queries. This is already really fast.
Also, may be making use of these bounding volume hierarchies for the weather 
module as an intermediate step might be a good idea?

Greetings
Mathias

> You mentioned earlier that a lot of the performance issues would
> disappear
> if we could probe the terrain 100 times faster.  I've been thinking about
> this for a while for ai traffic, skyop's moving map instrument, and
> weather.
>
> I'm thinking of storing some resolution of altitude data in the tile
> itself,
> making altitude queries very fast at the expense of a larger tile (in
> memory and disk).
>
> I'm just starting the work, but I would like any feedback on the idea.

I can maybe tell you what I need. Currently, Local Weather uses terrain
info in three ways:

1) a presampling routine gets gross features in the vicinity of the
aircraft, i.e. mean altitude, median altitude, max. altitude,... That's
now implemented by Torsten as C++ routine running in the background -  I
don't know details and don't know if it would benefit


2) convective clouds are placed based on both terrain type and elevation.
For convective (not any other) clouds, there is extra work running for
2-30 seconds (dependent on number, terrain detail,...) in the background
at the rate of ~12 geodinfo() calls per frame (geodinfo is a rather
expensive operation - more than 25 per frame are simply out of the
question).

3) for the 3rd generation cloud-terrain model currently under development,
there's an additional need to probe elevation (but not terrain type) for
many cloud types - curently using geodinfo() as well

2) and 3) are among the most computationally intensive processes Local
Weather runs, simply because geodinfo() is expensive (though perhaps not
the most garbage-collection triggering ones). If that could be made much
faster, tile setup could be accelerated dramatically.

I wouldn't actually need the exact altitude/terrain type at a given
location for weather - an approximation which gets me some elevation and
terrain type within, say, 100 m of the specified coordinates would be just
as good, especially if it comes faster. If it really accelerates by a
large margin, then it allows the setup of tiles to be essentially in a
single frame, so one can really restart the system without the user
noticing.

Cheers,

* Thorsten

2011/7/14 Mathias Fröhlich wrote:
> While being able to do a croase ground query on such a kind of regular grid
> might be beneficial for the weather module. I would prefer the ai module just
> using the already available bounding volume tree that is used for the main
> aircrafts elevation queries. This is already really fast.
> Also, may be making use of these bounding volume hierarchies for the weather
> module as an intermediate step might be a good idea?

Hi Mathias,

Presumably this is using the ground_cache rather code rather than the
scenery.get_elevation_m() code that the Nasal system uses to to get
geodinfo()

If so, I'll see if there's a sensible way to expose this over the
Nasal interface
as an alternative to the current geodinfo() routine.

Is there any reason not to simply use the ground_cache for the Nasal
geodinfo() routine? They both seem to make elevation and material
information available?


-Stuart

On Fri, Jul 29, 2011 at 3:18 PM, Stuart Buchanan wrote:
> 2011/7/14 Mathias Fröhlich wrote:
>> While being able to do a croase ground query on such a kind of regular grid
>> might be beneficial for the weather module. I would prefer the ai module just
>> using the already available bounding volume tree that is used for the main
>> aircrafts elevation queries. This is already really fast.
>> Also, may be making use of these bounding volume hierarchies for the weather
>> module as an intermediate step might be a good idea?
>
> Hi Mathias,
>
> Presumably this is using the ground_cache rather code rather than the
> scenery.get_elevation_m() code that the Nasal system uses to to get
> geodinfo()
>
> If so, I'll see if there's a sensible way to expose this over the
> Nasal interface
> as an alternative to the current geodinfo() routine.
>
> Is there any reason not to simply use the ground_cache for the Nasal
> geodinfo() routine? They both seem to make elevation and material
> information available?

So, I quickly wrote an alternative to the current Nasal  system geodinfo(),
using the groundcache instead of the  current scenery method.

I then tested both methods with a small piece of Nasal calling geodinfo
repeatedly across a degree of lat/lon at intervals of 0.01 degree in both
lat and lon. I.e. about 10,000 geodinfo calls.

Over multiple runs, the existing scenery based method took between
5.78 and 6.15 seconds. The groundcache version took between 0.38
and 0.5 seconds. So there seems to be about an order of magnitude
difference in performance between the two. In fact it may be more
than that, as there were a number of array operations
 taking place as well.

By inspection of a much smaller data set (0.1 degree steps), the differences
in altitude reported are < 1cm, and the materials returned are identical.

So, changing the existing geodinfo function for one that uses the groundcache
would appear to offer a significant improvement in performance, the
tradeoff being
presumably an increase in memory footprint due to use of the ground-cache.

For reference, the Nasal test code is copied below.

Comments?

-Stuart


------

var lat = getprop("/position/latitude-deg");
var lon = getprop("/position/longitude-deg");

var t = systime();

var alts = [];
var mats = [];
var count = 0;

print ("Time: " ~ t);

for (var i = 0; i < 1; i = i + 0.01) {
  for (var j = 0; j < 1; j = j + 0.01) {
    var geo_info = fastgeodinfo(lat + i, lon + j);
    if (geo_info != nil) {
       append(alts, geo_info[0]);

       if (geo_info[1] != nil) {
         append(mats, geo_info[1].names[0]);
        }
    }
  }
}

var d = systime();

print("Time: " ~ d);

print("Delta: " ~ (d - t));
print("Alts: " ~ size(alts) ~ " Mats: " ~ size(mats) ~ "\n");

> So, I quickly wrote an alternative to the current Nasal  system
> geodinfo(),
> using the groundcache instead of the  current scenery method.
(...)
> Comments?


You just made me a rather happy person :-) That seems like a sizeable
improvement in performance!

One question - do the two methods have the same range? I might call the
function for a spot 50 km away - dependent on visibility range, the
terrain there may already be loaded or not, and my code deals with both
possibilities - but if, say, the fast method would only work in a 10 km
radius around the aircraft, that might be a problem.

On 30 Jul 2011, at 20:31, Stuart Buchanan wrote:

> So, I quickly wrote an alternative to the current Nasal  system geodinfo(),
> using the groundcache instead of the  current scenery method.

I'm working on a new (C++) navigation display instrument, which I hope to add a proper EGPWS terrain display layer too - replacing Skyop's Nasal version, with his full and happy consent :)

I already reviewed the ground cache code, but I wan't sure how performant it would be, to scan the ND range (eg, 80, 160 or 320nm) at, say, 10 or 50m intervals. The problem is the EGPWS needs relative altitude to the aircraft, but in real-life the update isn't instantaneous, especially at longer ranges, so I expect to perform the scan incrementally over a few seconds.

James

Hi Stuart,
On 30 Jul 2011, at 21:31, Stuart Buchanan wrote:

> 
> So, I quickly wrote an alternative to the current Nasal  system geodinfo(),
> using the groundcache instead of the  current scenery method.
> 

This sounds very interesting: As far as I can tell, the AI system still makes use of scenery.get_elevation_m(), as shown in AIBase.cxx (line 516):

516 bool FGAIBase::getGroundElevationM(const SGGeod& pos, double& elev,
517                                    const SGMaterial** material) const {
518     return globals->get_scenery()->get_elevation_m(pos, elev, material,
519                                                    model.get());
520 }

I always thought that get_elevation_m would be using the ground cache itself, but apparently that is not the case? If so, it seems to me that changing the AI system to actually use the ground cache could yield another performance increase.

If you have any pointers on how this could be changed in an efficient way, I'd be happy to hear about it. :-)

Cheers,
Durk

On Mon, Aug 1, 2011 at 10:18 AM, Durk Talsma wrote:
> Hi Stuart,
> On 30 Jul 2011, at 21:31, Stuart Buchanan wrote:
>
>>
>> So, I quickly wrote an alternative to the current Nasal  system geodinfo(),
>> using the groundcache instead of the  current scenery method.
>>
>
> This sounds very interesting: As far as I can tell, the AI system still makes use of scenery.get_elevation_m(), as shown in AIBase.cxx (line 516):
>
> 516 bool FGAIBase::getGroundElevationM(const SGGeod& pos, double& elev,
> 517                                    const SGMaterial** material) const {
> 518     return globals->get_scenery()->get_elevation_m(pos, elev, material,
> 519                                                    model.get());
> 520 }
>
> I always thought that get_elevation_m would be using the ground cache itself, but apparently that is not the case? If so, it seems to me that changing the AI system to actually use the ground cache could yield another performance increase.
>
> If you have any pointers on how this could be changed in an efficient way, I'd be happy to hear about it. :-)

I'm looking to see whether we should just have a single way to query
the ground elevation using the groundcache, and use that everywhere.

However, the AI call you quote specifically excludes the model itself,
and I need to determine how to do that using the groundcache.

-Stuart

On Mon, Aug 1, 2011 at 9:48 AM, James Turner wrote:
>
> On 30 Jul 2011, at 20:31, Stuart Buchanan wrote:
>
>> So, I quickly wrote an alternative to the current Nasal  system geodinfo(),
>> using the groundcache instead of the  current scenery method.
>
> I'm working on a new (C++) navigation display instrument, which I hope to add a proper EGPWS terrain display layer too - replacing Skyop's Nasal version, with his full and happy consent :)
>
> I already reviewed the ground cache code, but I wan't sure how performant it would be, to scan the ND range (eg, 80, 160 or 320nm) at, say, 10 or 50m intervals. The problem is the EGPWS needs relative altitude to the aircraft, but in real-life the update isn't instantaneous, especially at longer ranges, so I expect to perform the scan incrementally over a few seconds.

In both cases, are you not going to be limited by what tiles have been loaded?

I did a further test just going in a straight line for 4 degrees in
0.001 degree increments, but only picked up some 400 points before
both methods just start returning 0 elevation due to there being no
tile loaded. FTR, I was seeing around 0.8s to make the 4000 queries
using the old method, 0.045s with the new, so it's significantly
faster in that case as well.

-Stuart

On 1 Aug 2011, at 22:35, Stuart Buchanan wrote:

> In both cases, are you not going to be limited by what tiles have been loaded?

Yes - I have wondered about separately loading the BTG files, but that seems like a world of pain. In the first instance, simply having the tiles loaded in the cache would be a reasonable start.

James

On 02.08.2011 00:30, James Turner wrote:
> Yes - I have wondered about separately loading the BTG files, but
> that seems like a world of pain. In the first instance, simply having
> the tiles loaded in the cache would be a reasonable start.

The tile manager is capable of satisfying multiple requests. Anyone can 
give it a position, range and a timeout. It will then try to load all 
tiles in the range specified. And it will stop loading them after the 
timeout - unless you have updated the request with a new timeout. So you 
could tell it every 5 seconds that you're interested in a certain area 
around the aircraft, and use a timeout of 5,01 seconds. A matter of 
memory and loading speed though ;-).

cheers,
Thorsten

Hi,

On Thursday, July 14, 2011 11:04:50 thorsten.i.renk@... wrote:
> I can maybe tell you what I need. Currently, Local Weather uses terrain
> info in three ways:
> 1) a presampling routine gets gross features in the vicinity of the
> aircraft, i.e. mean altitude, median altitude, max. altitude,... That's
> now implemented by Torsten as C++ routine running in the background -  I
> don't know details and don't know if it would benefit
> 
> 
> 2) convective clouds are placed based on both terrain type and elevation.
> For convective (not any other) clouds, there is extra work running for
> 2-30 seconds (dependent on number, terrain detail,...) in the background
> at the rate of ~12 geodinfo() calls per frame (geodinfo is a rather
> expensive operation - more than 25 per frame are simply out of the
> question).
> 
> 3) for the 3rd generation cloud-terrain model currently under development,
> there's an additional need to probe elevation (but not terrain type) for
> many cloud types - curently using geodinfo() as well
> 
> 2) and 3) are among the most computationally intensive processes Local
> Weather runs, simply because geodinfo() is expensive (though perhaps not
> the most garbage-collection triggering ones). If that could be made much
> faster, tile setup could be accelerated dramatically.
> 
> I wouldn't actually need the exact altitude/terrain type at a given
> location for weather - an approximation which gets me some elevation and
> terrain type within, say, 100 m of the specified coordinates would be just
> as good, especially if it comes faster. If it really accelerates by a
> large margin, then it allows the setup of tiles to be essentially in a
> single frame, so one can really restart the system without the user
> noticing.

Thanks for explaining.
I will tell you what the ground cache is and how this relates to the scene 
graph:

We have currently all our environment in the scene graph. For fast rendering 
it is required to have huge leaf nodes containing the actual geometry in the 
scene graph. Then there are these intersection tests with that special case 
ground elevation lookup. For this purpose you would like to have also a 
graph/tree like structure but with leaf nodes containing exactly one geometry 
primitive. One of these fast intersection structures is a bounding volume 
tree. Such a tree has usually leaf nodes containing just one geometry 
primitive. Which is just the opposite of what the gpu needs for rendering.

So I decoupled these two structures somehow. I have put bv-trees of geometry 
into the userdata field of the scenegraph. So for the high level operations 
like tile loading, the scenery paging is used to load and get rid of the bv-
trees. The whole intersectable scenery should be there in this form.
Using this, you could already speed up ground queries by using these bv-tree 
leafs for intersection test leaf traversal.

But for the actual aircraft I wanted to be able to do up to the order of 
100-1000 intersection tests per timestep. To get that sufficiently fast, I built 
that ground cache:
That is just a bv-tree of the near environment of the aircraft. This is 
computed by traversing the scenegraph, and collecting hose subparts of the 
leaf bv-trees that are within a few 10 to 100 meters away from the aircraft. 
Just as much to cover the whole possible travel distance given the current 
speed and position. Queries into this structure are extremely fast (~1e-6s - 
1e-5s) on my development notebook. Building this substructure is usually about 
1e-3s depending on its size. If there is no geometry in the groundcache, since 
we are flying too high, there is one single elevation query into the whole 
scenegraph that give the agl to give reasonable values.
Special care must be taken for moving geometries like the Aircraft carrier.

Ground type information is stored in the scenegraph and referenced by the bv-
tree. So, no matter how you query, if you do not have the ground type 
information, there is an implementation just not fetching it.

This means using the ground cache is only valid near the actual aircraft, 
where near means a few 10 meters. And no, please do not increase this ground 
cache radius of the aircraft for the weather.

What you can do for the weather is to traverse the bv-tree nodes instead of 
the GPU optimized scenery nodes for ground queries. I guess this is the 
fastest (implementation wise fastest) approach to get something you need.

I still like the idea - only for the weather module - to have a cartesian grid 
as a 'weather ground cache'. To me this sounds like something which provides a 
litte more croase, but sufficieltly good approximations to scenery altitude. You 
could also average ground type information for your needs at the grid nodes - 
whereas you would just get singular ground types for traditional scene 
queries. Sure, this is much more work to set this up.

In the future, I hope to have a completely independent weather module using 
the HLA stuff that runs in an own process/thread. So, at that time you might be 
able to do more sophisticated stuff. May be it will then be possible to do a 
full cfd for subparts of the scene. That might be a good thing for a glider 
scene where you might want to have a more detailed fluid behaviour ...

Greetings

Mathias

Hi,

On Saturday, July 30, 2011 21:31:37 Stuart Buchanan wrote:
> > Hi Mathias,
Sorry not to anser in time ...


> > Presumably this is using the ground_cache rather code rather than the
> > scenery.get_elevation_m() code that the Nasal system uses to to get
> > geodinfo()
> > 
> > If so, I'll see if there's a sensible way to expose this over the
> > Nasal interface
> > as an alternative to the current geodinfo() routine.
> > 
> > Is there any reason not to simply use the ground_cache for the Nasal
> > geodinfo() routine? They both seem to make elevation and material
> > information available?
Do not use the groundcache as such, as it is only valid near the aircraft. But 
you can use the the bv-trees in the userdata nodes for queries.
This should be perfectly ok.

> Comments?
I have not checked the git source tree.
But what are you doing at the c++ level?

Mathias

Durk,

On Monday, August 01, 2011 11:18:02 Durk Talsma wrote:
> On 30 Jul 2011, at 21:31, Stuart Buchanan wrote:
> > So, I quickly wrote an alternative to the current Nasal  system
> > geodinfo(), using the groundcache instead of the  current scenery
> > method.
> 
> This sounds very interesting: As far as I can tell, the AI system still
> makes use of scenery.get_elevation_m(), as shown in AIBase.cxx (line 516):
> 
> 516 bool FGAIBase::getGroundElevationM(const SGGeod& pos, double& elev,
> 517                                    const SGMaterial** material) const {
> 518     return globals->get_scenery()->get_elevation_m(pos, elev, material,
> 519                                                    model.get());
> 520 }
> 
> I always thought that get_elevation_m would be using the ground cache
> itself, but apparently that is not the case? If so, it seems to me that
> changing the AI system to actually use the ground cache could yield
> another performance increase.
> 
> If you have any pointers on how this could be changed in an efficient way,
> I'd be happy to hear about it. :-)

Ok, the lengthy explanation hof the ground cache works in the previous mail 
might help to understand what you can do.

I think that using the bv-trees and the whole scenegraph would make sense for 
the AI module. That is basically the same than what I would suggest as a 
simple solution for the weather module.

Alternatively I can imagine to have each AI model have its own groundcache. In 
this case somehow bigger than the very short time groundcache of the aircraft. 
May be big enough to cover the movement of one AI model for some seconds. Then 
during these few seconds - or as long as the aircraft is just sufficiently 
inside the cache - make queries into the local cache.

Also, I hope to get that improoved with a seperate AI component in a HLA 
federation.

Greetings

Mathias

Stuart,

On Monday, August 01, 2011 21:51:16 Stuart Buchanan wrote:
> I'm looking to see whether we should just have a single way to query
> the ground elevation using the groundcache, and use that everywhere.
See the lenghty explanation in the response to the weather system.

> However, the AI call you quote specifically excludes the model itself,
> and I need to determine how to do that using the groundcache.

That is more or less simple:
You need to traverse the scenegraph group nodes anyway. If you would just know 
the pointer to the own aircrafts top level group node, you can skip the own 
aircrafts subtree ...
This holds for building a AI local ground cache for each AI model as well as 
for a single ground query using the bv-tree leafs.

Greetings

Mathias

ThorstenB wrote:

> 
> On 02.08.2011 00:30, James Turner wrote:
> > Yes - I have wondered about separately loading the BTG files, but
> > that seems like a world of pain. In the first instance, simply having
> > the tiles loaded in the cache would be a reasonable start.
> 
> The tile manager is capable of satisfying multiple requests. Anyone can
> give it a position, range and a timeout. It will then try to load all
> tiles in the range specified. And it will stop loading them after the
> timeout - unless you have updated the request with a new timeout. So you
> could tell it every 5 seconds that you're interested in a certain area
> around the aircraft, and use a timeout of 5,01 seconds. A matter of
> memory and loading speed though ;-).
> 

Meanwhile - at the cheap end of the market, Emilian and I have cleaned up
most of the textures, made some of the AC3D models single-sided, reordered
the objects, provided unique names, and converted everything to .dds.

This has much improved loading/unloading, and given a much better appearance
when panning the view. Flicking on/off of clouds has pretty much been
eliminated. There is still a bit of a problem reloading in fly-by view, but
it is much improved. Some weirdness near the zenith has also been fixed.

Available here:

https://gitorious.org/weather_textures/weather_textures

I have managed to break non-detailed clouds along the way, but hey - you
can't have everything.

All in all a worthwhile exercise, with some tidying up still to do.

When you experts have fixed up the ground cache - we might have something of
a success. 

Vivian

> Meanwhile - at the cheap end of the market, Emilian and I have cleaned up
> most of the textures, made some of the AC3D models single-sided,
> reordered
> the objects, provided unique names, and converted everything to .dds.
>
> This has much improved loading/unloading, and given a much better
> appearance
> when panning the view. Flicking on/off of clouds has pretty much been
> eliminated. There is still a bit of a problem reloading in fly-by view,
> but
> it is much improved. Some weirdness near the zenith has also been fixed.
>
> Available here:
>
> https://gitorious.org/weather_textures/weather_textures
>
> I have managed to break non-detailed clouds along the way, but hey - you
> can't have everything.

Thanks to both of you for the work! I had some trouble getting it
yesterday (lousy internet connection...) - I'm getting the package as I
write this mail and will have a look soon.

Maybe I can also figure out the problem with the simple Cu clouds - I'm
not attached to them as such, but they are handy for the edges of Sc
layers... so I don't really want to lose them.

* Thorsten

Thorsten wrote:

 
> > Meanwhile - at the cheap end of the market, Emilian and I have cleaned
> up
> > most of the textures, made some of the AC3D models single-sided,
> > reordered
> > the objects, provided unique names, and converted everything to .dds.
> >
> > This has much improved loading/unloading, and given a much better
> > appearance
> > when panning the view. Flicking on/off of clouds has pretty much been
> > eliminated. There is still a bit of a problem reloading in fly-by view,
> > but
> > it is much improved. Some weirdness near the zenith has also been fixed.
> >
> > Available here:
> >
> > https://gitorious.org/weather_textures/weather_textures
> >
> > I have managed to break non-detailed clouds along the way, but hey - you
> > can't have everything.
> 
> Thanks to both of you for the work! I had some trouble getting it
> yesterday (lousy internet connection...) - I'm getting the package as I
> write this mail and will have a look soon.
> 
> Maybe I can also figure out the problem with the simple Cu clouds - I'm
> not attached to them as such, but they are handy for the edges of Sc
> layers... so I don't really want to lose them.
> 

That bug has been fixed - I finally found the one remaining "rect" that was
causing the problems.

This work is not the answer to the maiden's prayer - the gains are marginal
- but worth having. Here, Local Weather is usable providing none of the more
fancy features are enabled, and I ignore the Fly-By view. I'm very taken
with the layered clouds. Some Cu:

http://imageshack.us/photo/my-images/837/fgfsscreen001.jpg/

Vivian

2011/8/2 Mathias Fröhlich wrote:
<snip>
> So I decoupled these two structures somehow. I have put bv-trees of geometry
> into the userdata field of the scenegraph. So for the high level operations
> like tile loading, the scenery paging is used to load and get rid of the bv-
> trees. The whole intersectable scenery should be there in this form.
> Using this, you could already speed up ground queries by using these bv-tree
> leafs for intersection test leaf traversal.
>
> But for the actual aircraft I wanted to be able to do up to the order of
> 100-1000 intersection tests per timestep. To get that sufficiently fast, I built
> that ground cache:

<snip>

> This means using the ground cache is only valid near the actual aircraft,
> where near means a few 10 meters. And no, please do not increase this ground
> cache radius of the aircraft for the weather.
>
> What you can do for the weather is to traverse the bv-tree nodes instead of
> the GPU optimized scenery nodes for ground queries. I guess this is the
> fastest (implementation wise fastest) approach to get something you need.

Thanks very much for the detailed explanation. That makes a lot of things much
clearer!

>From a quick code read, it looks like the scenery.cxx get_elevation_m() function
does not use the bv-tree.

I think the reason I have seen much higher performance using the "ground_cache"
is that the flight.cxx method uses the BVH tree if it doesn't get a scenery hit.

Therefore the solution may be to change the scenery.cxx version to use the BVH
tree if possible. That should provide significant performance gains. I
will investigate...

-Stuart

2011/8/2 Mathias Fröhlich wrote:
> So I decoupled these two structures somehow. I have put bv-trees of geometry
> into the userdata field of the scenegraph. So for the high level operations
> like tile loading, the scenery paging is used to load and get rid of the bv-
> trees. The whole intersectable scenery should be there in this form.
> Using this, you could already speed up ground queries by using these bv-tree
> leafs for intersection test leaf traversal.

I've spent quite a while looking at the scenery code to see where the
bv-trees are being
generated and saved to the SGSceneUserData, but so far the only place
I've found that
calls the BoundingVolumeBuildVisitor is the groundcache and ModelRegistry.

Furthermore, calling getBVHNode() on the SGSceneUserData associated with a tile
nevery returns anything.

Am I just missing something completely here, or is there coding
required to generate the
BVH trees for the scenery tiles themselves (presumably based on the
groundcache code?)

-Stuart

Hi Stuart,

I have checked in a version of the ground intersection routines that
just use the bvh trees in the scenegraph.
This should improove the run time behaviour for the ground queries.
That should be even faster than the variant you tried since it avoids the 
extra computations/allocations that are required for building up a ground 
cache agound this point.

If you experience any problems with this change please tell me.

Greetings

Mathias

2011/8/7 Mathias Fröhlich:
> I have checked in a version of the ground intersection routines that
> just use the bvh trees in the scenegraph.
> This should improove the run time behaviour for the ground queries.
> That should be even faster than the variant you tried since it avoids the
> extra computations/allocations that are required for building up a ground
> cache agound this point.
>
> If you experience any problems with this change please tell me.

Thank you very much for doing this. It is indeed much faster than the previous
queries, and much cleaner code than I think I could have written.

I'm sure you've made Thorsten very happy indeed!

-Stuart

Hi,

On Sunday, August 07, 2011 12:32:36 Stuart Buchanan wrote:
> 2011/8/7 Mathias Fröhlich:
> > I have checked in a version of the ground intersection routines that
> > just use the bvh trees in the scenegraph.
> > This should improove the run time behaviour for the ground queries.
> > That should be even faster than the variant you tried since it avoids the
> > extra computations/allocations that are required for building up a ground
> > cache agound this point.
> > 
> > If you experience any problems with this change please tell me.
> 
> Thank you very much for doing this. It is indeed much faster than the
> previous queries, and much cleaner code than I think I could have written.
Hmm, thanks.

I have now seen that this has overlapped a question from yours form yesterday 
evening. I just got up today morning, took a look outside - still no summer in 
mid europe - and decided to hack something. Then I just thought that this 
change might be good to have...
:)

Did you really measure again how much faster this is? Just some average rule 
of thumb?

Greetings

Mathias

2011/8/7 Mathias Fröhlich wrote:
> I have now seen that this has overlapped a question from yours form yesterday
> evening. I just got up today morning, took a look outside - still no summer in
> mid europe - and decided to hack something. Then I just thought that this
> change might be good to have...
> :)

Same problem in Scotland :)

> Did you really measure again how much faster this is? Just some average rule
> of thumb?

Running the same test as described above (Nasal geodinfo() called in 0.01 steps
across a 1x1 degree area) in the same place, I got the following
results from multiple
runs of the code.

Old scenery.cxx method: 5.78s - 6.15s
Groundcache hack: 0.38 - 0.5s
New scenery.cxx method: 0.1s - 0.12s

The tests were run at different times, so there may be some environmental
differences based on what else was running on my laptop etc.

Nevertheless, that's a massive performance improvement over the old method,
and a significant improvement over the groundcache hack I wrote.

-Stuart

2011/8/7 Mathias Fröhlich wrote:
>
> Hi Stuart,
>
> I have checked in a version of the ground intersection routines that
> just use the bvh trees in the scenegraph.
> This should improove the run time behaviour for the ground queries.
> That should be even faster than the variant you tried since it avoids the
> extra computations/allocations that are required for building up a ground
> cache agound this point.
>
> If you experience any problems with this change please tell me.

I think this may have broken YASim aircraft. I've tried a selection (dc3, dc-3,
a4f) and they all start at 36,000ft altitude with 0 IAS.

I suspect it might be an initialization order issue,

I've raised it as bug 397
(http://code.google.com/p/flightgear-bugs/issues/detail?id=397)

-Stuart

Hi,

On Sunday, August 07, 2011 20:46:46 Stuart Buchanan wrote:
> I think this may have broken YASim aircraft. I've tried a selection (dc3,
> dc-3, a4f) and they all start at 36,000ft altitude with 0 IAS.
> 
> I suspect it might be an initialization order issue,
> 
> I've raised it as bug 397
> (http://code.google.com/p/flightgear-bugs/issues/detail?id=397)
I will look into this. Today evening.

Mathias

I wrote

> 
> Thorsten wrote:
> 
> 
> > > Meanwhile - at the cheap end of the market, Emilian and I have cleaned
> > up
> > > most of the textures, made some of the AC3D models single-sided,
> > > reordered
> > > the objects, provided unique names, and converted everything to .dds.
> > >
> > > This has much improved loading/unloading, and given a much better
> > > appearance
> > > when panning the view. Flicking on/off of clouds has pretty much been
> > > eliminated. There is still a bit of a problem reloading in fly-by
> view,
> > > but
> > > it is much improved. Some weirdness near the zenith has also been
> fixed.
> > >
> > > Available here:
> > >
> > > https://gitorious.org/weather_textures/weather_textures
> > >
> > > I have managed to break non-detailed clouds along the way, but hey -
> you
> > > can't have everything.
> >
> > Thanks to both of you for the work! I had some trouble getting it
> > yesterday (lousy internet connection...) - I'm getting the package as I
> > write this mail and will have a look soon.
> >
> > Maybe I can also figure out the problem with the simple Cu clouds - I'm
> > not attached to them as such, but they are handy for the edges of Sc
> > layers... so I don't really want to lose them.
> >
> 
> That bug has been fixed - I finally found the one remaining "rect" that
> was
> causing the problems.
> 
> This work is not the answer to the maiden's prayer - the gains are
> marginal
> - but worth having. Here, Local Weather is usable providing none of the
> more
> fancy features are enabled, and I ignore the Fly-By view. I'm very taken
> with the layered clouds. Some Cu:
> 
> http://imageshack.us/photo/my-images/837/fgfsscreen001.jpg/
> 

Unless there are any objections, I intend to push Emilian's and my work to
Git later today. I appreciate that it will likely be overtaken by events,
and hopefully will be only be a short term fix.

Vivian

> Unless there are any objections, I intend to push Emilian's and my work
> to Git later today.

Has been working fine for me the last days - no problems with dds
textures, and I've been using your reworked texture sheets as basis to
generate the regular m x n structures needed for Stuart's system.

> I appreciate that it will likely be overtaken by events,
> and hopefully will be only be a short term fix.

It may still be a while until everything is fully in place - I can
generate and remove various clouds by now (the easier ones...), but I
don't really have long-range functionality yet, no cloud evolution in
altitude or texture, still some problems with shading - expect a month or
two for a proper release at least, althoug maybe some experimental version
earlier.

* Thorsten