Menu
▾
▴
Re: [geowebcache-users] GeoWebCache directory/file name structure
|
From: samclemmens <geo...@gm...> - 2010-06-18 11:23:34
|
Hi Arne, Tim is processing the data in EC2 per your recommendation above. He estimates that it will take 30 hours to process national coverage for Great Britain and then another 15 hours to create the spatial indexes... In the meantime, I have a question about the parameters in the FilePathGenerator class. (Some of these are obvious, but I just want to confirm.) Can you tell me what the following are? - prefix - tileIndex - gridSetId - parameters_id - x (northing?) - y (easting?) - gridSetStr By the way, have you guys considered creating an AMI with the new OpenGeo stack? I got most of the individual pieces working on a Fedora instance last month, but it would have been a heck of a lot easier to just fire up an AMI... Thanks, Peter TMartin wrote: > > Hi Arne > > Will give this a go at the end of the week and will let you know how we > get on > > thanks > > Tim > > > > Arne Kepp-2 wrote: >> >> Hi Tim >> >> I am saying that if you had defined the whole array of resolutions in >> GWC then you would have gotten a reasonably good directory structure, >> decimating the tiles differently due to z=10 instead of z=0. >> >> Think of the directories as buckets, and a larger Z makes each directory >> accept a larger range of X and Y values. At the same time it keeps tiles >> that are close together in the same directory. Peter seems to have gone >> through the formula in detail, so perhaps it's easier for him to explain >> in person than it is for me via email. >> >> Use this (you'll have to clear the cache, sorry) >> >> <resolutions> >> <double>2000</double> >> <double>1000</double> >> <double>500</double> >> <double>200</double> >> <double>100</double> >> <double>50</double> >> <double>25</double> >> <double>10</double> >> <double>5</double> >> <double>2</double> >> <double>1</double> >> </resolutions> >> >> >> and then just seed the last level and take that with you. The other >> levels probably wont take long anyway, depending on how you've styled it. >> >> -Arne >> >> >> >> On 06/14/2010 04:28 PM, TMartin wrote: >>> Hi Arne >>> >>> In our OpenSpace API we have 10 zoom levels and their resolutions. >>> >>> 10 Streetview 1m >>> >>> 9 Streetview Resampled 2m >>> >>> 8 1:50 000 5m >>> >>> 7 1:50 000 Resampled 10m >>> >>> 6 1:250 000 25m >>> >>> 5 1:250 000 Resampled 50m >>> >>> 4 MiniScale 100m >>> >>> 3 MiniScale Resampled 200m >>> >>> 2 Overview 500m >>> >>> 1 Overview Resampled 1000m >>> >>> 0 Overview 2000m >>> >>> These are already raster tiles and are chopped into either 200x200 pixel >>> tiles or 250x250 pixel tiles depending on their original tile size. >>> >>> When OpenSpace was built we used our own tile naming convention and >>> directory structure because there were no others in existence. >>> >>> We are now wanting to create tiles from a vector dataset to replace >>> raster >>> dataset at zoom levels 10 and 9 (resolution 1m and 2m) >>> >>> So we have PostGIS with the data in, Geoserver 2.0.2, style sheets >>> (SLDs) >>> and want to use Geowebcache to create the tiles in EPSG 27700 (obviously >>> british national grid ;) >>> >>> After spending sometime with a calculator i have worked out that we need >>> 14,560,000 250x250 pixel tiles to cover GB, with coordinates 0, 0, >>> 700000, >>> 1300000 at 1m resolution. >>> >>> This is the configuration file I have been using, and it works and >>> creates >>> tiles. >>> >>> >>> <?xml version="1.0" encoding="utf-8"?> >>> <gwcConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> >>> xsi:noNamespaceSchemaLocation="http://geowebcache.org/schema/1.2.0/geowebcache.xsd" >>> xmlns="http://geowebcache.org/schema/1.2.0"> >>> <version>1.2.0</version> >>> <backendTimeout>300</backendTimeout> >>> >>> <gridSets> >>> <!-- >>> String name; >>> SRS srs; >>> BoundingBox extent; >>> double[] resolutions; >>> double[] scales; >>> String[] scaleNames; >>> Integer levels; >>> Integer tileHeight; >>> Integer tileWidth; >>> --> >>> >>> >>> >>> <gridSet> >>> <name>osgb_vml</name> >>> <srs><number>27700</number></srs> >>> <extent> >>> <coords> >>> <double>0</double> >>> <double>0</double> >>> <double>700000</double> >>> <double>1300000</double> >>> </coords> >>> </extent> >>> <alignTopLeft>false</alignTopLeft> >>> <resolutions> >>> <double>1</double> >>> </resolutions> >>> <tileHeight>250</tileHeight> >>> <tileWidth>250</tileWidth> >>> </gridSet> >>> >>> >>> </gridSets> >>> <layers> >>> >>> >>> <wmsLayer> >>> <name>osgb:vml/name> >>> <gridSubsets> >>> <gridSubset> >>> <gridSetName>osgb_vml</gridSetName> >>> </gridSubset> >>> </gridSubsets> >>> >>> <wmsUrl><string>http://localhost:8080/geoserver/wms</string></wmsUrl> >>> >>> <wmsLayers>osgb:areas,osgb:lines,osgb:roadclines,osgb:text</wmsLayers> >>> <!-- OPTIONAL The metatiling factors used for this layer >>> If not specified, 3x3 metatiling is used for image formats --> >>> <metaWidthHeight><int>1</int><int>1</int></metaWidthHeight> >>> <bgColor>0xFFFFFF</bgColor> >>> </wmsLayer> >>> >>> </layers> >>> </gwcConfiguration> >>> >>> >>> So when i seed this layer and go to the setup page i only the option of >>> seeding between zoom level 00 and finish zoom level 00. >>> >>> Which is fine. >>> >>> The problem as Pete mentioned above is the shear number of directories >>> that >>> only have 4 tiles in each folder. >>> >>> I hope that gives a better picture of what we are up do. I appreciate >>> its an >>> extreme case but when we get it sorted we will post up the solution and >>> the >>> timeframes it took. >>> >>> cheers >>> >>> Tim >>> >>> >>> >>> >>> >>> Arne Kepp-2 wrote: >>> >>>> So the plot thickens :) >>>> >>>> You mentioned in another email that you were just seeding one >>>> zoomlevel. >>>> So I guess you also made a grid set configuration with very few levels, >>>> perhaps just 2? >>>> >>>> This isn't useful for most GWC users, you need a tile pyramid with many >>>> levels sot hat people can gradually zoom in. I am guessing that for >>>> OpenSpace you'll be subsampling these tiles to make up intermediate >>>> resolutions. >>>> >>>> So this becomes a very extreme case that the default directory >>>> structure >>>> was definitely not designed to handle, normally you only have 1 to 16 >>>> tiles at zoomlevel 0 , so by the time you get down to 16 million tiles >>>> (z=12 or so) they should be spread across 4000 directories with 4000 >>>> files each. >>>> >>>> If you have a thought-out directory structure for your application, I >>>> would implement that as a new BlobStore, because your needs will be >>>> different from the normal GWC user. Alternatively, you could use the >>>> existing tile structure and just define a few resolutions you will not >>>> use, so that the level with 16 million tiles is z=12 or so. >>>> >>>> Note that you need to clear the cache (delete the directory) if you >>>> change the grid set definition. >>>> >>>> -Arne >>>> >>>> >>>> On 06/10/2010 10:05 PM, TMartin wrote: >>>> >>>>> Hi Arne >>>>> >>>>> Tim here. Pete and I are both working on the same issue here at OS. >>>>> >>>>> In each directory there are only 4 tiles of 250x250pixels and 1metre >>>>> resolution as specified in our geowebcache configuration xml document. >>>>> >>>>> If thats the case we may end up with over 3 million directories for >>>>> the >>>>> 14,560,000 tiles we are creating. >>>>> >>>>> I will try and set it up so you can take a look at if you are >>>>> interested? >>>>> >>>>> Tim >>>>> >>>>> >>>>> >>>>> Arne Kepp-2 wrote: >>>>> >>>>> >>>>>> It's a compromise, and this part of the code is definitely made >>>>>> easily >>>>>> pluggable so that one can use a different structures for special >>>>>> needs. >>>>>> I was expecting that we would have a few different implementations by >>>>>> now, but apparently not. >>>>>> >>>>>> Every directory level requires a node to be looked up (move the disk >>>>>> head), so too many directories is not good either. Within a directory >>>>>> there is a proper index. Listing the content of the cache is >>>>>> generally >>>>>> not recommended, I don't know of a single filesystem that is >>>>>> optimized >>>>>> for that and most utilities can't handle something of this size. >>>>>> "find" >>>>>> is usually ok. >>>>>> >>>>>> But 400 000 directories.... Unless you've discovered a bug, you >>>>>> should >>>>>> have at least 80 billion tiles? I wouldn't use a conventional >>>>>> filesystem for that, and on a fast machine (100 tiles per second) it >>>>>> would take 25 years to build a cache of that size. >>>>>> >>>>>> -Arne >>>>>> >>>>>> >>>>>> On 06/10/2010 11:22 AM, samclemmens wrote: >>>>>> >>>>>> >>>>>>> Cheers, Arne! >>>>>>> >>>>>>> I'm curious; what was the rationale for creating a directory >>>>>>> structure >>>>>>> with >>>>>>> just one level? We processed a data set earlier this week, which >>>>>>> resulted >>>>>>> in 400,000 directories. When I tried to an "ls", my PuTTY console >>>>>>> froze... >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> >>>>>>> Arne Kepp-2 wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> 1) "zoom level". For the default EPSG:900913 grid z=0 means the >>>>>>>> whole >>>>>>>> world in 4 tiles >>>>>>>> >>>>>>>> 2) Hm,,, good point. I dont think I went back and updated that wiki >>>>>>>> page, it was not more than a scratchpad for sharing ideas. The >>>>>>>> actual >>>>>>>> code is here[1], the difference is that the bit shifting starts >>>>>>>> with >>>>>>>> 2, >>>>>>>> effectively making it 2^( 1 + ( z / 2 )). And there's some >>>>>>>> zero-padding. >>>>>>>> >>>>>>>> 3) The catch is that you need to know a little bit of Java, but if >>>>>>>> you >>>>>>>> do then it's potentially straightforward. It depends what the >>>>>>>> OpenSpace >>>>>>>> structure is based on, and whether we have similar tokens in GWC. >>>>>>>> >>>>>>>> blobstore/file contains an implementation of BlobStore. You can >>>>>>>> copy >>>>>>>> the >>>>>>>> whole thing to a new package, and then make the FilePathGenerator >>>>>>>> output >>>>>>>> the structure you want. It is linked into the application using >>>>>>>> Spring, >>>>>>>> WEB-INF/geowebcache-core-context.xml , if you are working off >>>>>>>> trunk. I >>>>>>>> would guess that you also want a particular GridSet to go with >>>>>>>> that, >>>>>>>> to >>>>>>>> match the resolutions and origin used by OpenSpace. >>>>>>>> >>>>>>>> It's possible someone has already done it, but so far I have not >>>>>>>> heard >>>>>>>> about it. >>>>>>>> >>>>>>>> -Arne >>>>>>>> >>>>>>>> 1: >>>>>>>> http://geowebcache.org/trac/browser/trunk/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/file/FilePathGenerator.java >>>>>>>> >>>>>>>> >>>>>>>> On 06/08/2010 05:33 PM, samclemmens wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm relatively new to GeoWebCache...and I'm a bit puzzled by the >>>>>>>>> directory/file name structure that is produced. I have a few >>>>>>>>> questions, >>>>>>>>> namely: >>>>>>>>> >>>>>>>>> 1. The documentation states that the naming convention is: >>>>>>>>> layername/projection_z/[x/(2^(z/2))]_[y/(2^(z/2))]/x_y.extension >>>>>>>>> What >>>>>>>>> is >>>>>>>>> the z value? >>>>>>>>> >>>>>>>>> 2. Given the sample output >>>>>>>>> .\gwc\roads\EPSG_900913_12\006_019\000848_002540.png, I assume >>>>>>>>> that >>>>>>>>> z=12, >>>>>>>>> x=848, and y=2540. Obviously the above formula won't produce >>>>>>>>> 006_019 >>>>>>>>> based >>>>>>>>> on these numbers. What conversion is required? >>>>>>>>> >>>>>>>>> 3. Is it possible to change the directory/file name format? I'd >>>>>>>>> like >>>>>>>>> to >>>>>>>>> create a data set that can be used by Ordnance Survey OpenSpace, >>>>>>>>> which >>>>>>>>> uses >>>>>>>>> a different structure based on the British National Grid. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> Peter >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>>>>>>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>>>>>>> lucky parental unit. See the prize list and enter to win: >>>>>>>> http://p.sf.net/sfu/thinkgeek-promo >>>>>>>> _______________________________________________ >>>>>>>> Geowebcache-users mailing list >>>>>>>> Geo...@li... >>>>>>>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>>>>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>>>>> lucky parental unit. See the prize list and enter to win: >>>>>> http://p.sf.net/sfu/thinkgeek-promo >>>>>> _______________________________________________ >>>>>> Geowebcache-users mailing list >>>>>> Geo...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>>> lucky parental unit. See the prize list and enter to win: >>>> http://p.sf.net/sfu/thinkgeek-promo >>>> _______________________________________________ >>>> Geowebcache-users mailing list >>>> Geo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>>> >>>> >>>> >>> >> >> >> ------------------------------------------------------------------------------ >> ThinkGeek and WIRED's GeekDad team up for the Ultimate >> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >> lucky parental unit. See the prize list and enter to win: >> http://p.sf.net/sfu/thinkgeek-promo >> _______________________________________________ >> Geowebcache-users mailing list >> Geo...@li... >> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >> >> > > -- View this message in context: http://old.nabble.com/GeoWebCache-directory-file-name-structure-tp28819540p28924991.html Sent from the GeoWebCache Users mailing list archive at Nabble.com. |
