Menu
▾
▴
Re: [geowebcache-users] GeoWebCache directory/file name structure
|
From: TMartin <Tim...@or...> - 2010-06-15 09:23:18
|
Hi Arne Will give this a go at the end of the week and will let you know how we get on thanks Tim Arne Kepp-2 wrote: > > Hi Tim > > I am saying that if you had defined the whole array of resolutions in > GWC then you would have gotten a reasonably good directory structure, > decimating the tiles differently due to z=10 instead of z=0. > > Think of the directories as buckets, and a larger Z makes each directory > accept a larger range of X and Y values. At the same time it keeps tiles > that are close together in the same directory. Peter seems to have gone > through the formula in detail, so perhaps it's easier for him to explain > in person than it is for me via email. > > Use this (you'll have to clear the cache, sorry) > > <resolutions> > <double>2000</double> > <double>1000</double> > <double>500</double> > <double>200</double> > <double>100</double> > <double>50</double> > <double>25</double> > <double>10</double> > <double>5</double> > <double>2</double> > <double>1</double> > </resolutions> > > > and then just seed the last level and take that with you. The other > levels probably wont take long anyway, depending on how you've styled it. > > -Arne > > > > On 06/14/2010 04:28 PM, TMartin wrote: >> Hi Arne >> >> In our OpenSpace API we have 10 zoom levels and their resolutions. >> >> 10 Streetview 1m >> >> 9 Streetview Resampled 2m >> >> 8 1:50 000 5m >> >> 7 1:50 000 Resampled 10m >> >> 6 1:250 000 25m >> >> 5 1:250 000 Resampled 50m >> >> 4 MiniScale 100m >> >> 3 MiniScale Resampled 200m >> >> 2 Overview 500m >> >> 1 Overview Resampled 1000m >> >> 0 Overview 2000m >> >> These are already raster tiles and are chopped into either 200x200 pixel >> tiles or 250x250 pixel tiles depending on their original tile size. >> >> When OpenSpace was built we used our own tile naming convention and >> directory structure because there were no others in existence. >> >> We are now wanting to create tiles from a vector dataset to replace >> raster >> dataset at zoom levels 10 and 9 (resolution 1m and 2m) >> >> So we have PostGIS with the data in, Geoserver 2.0.2, style sheets (SLDs) >> and want to use Geowebcache to create the tiles in EPSG 27700 (obviously >> british national grid ;) >> >> After spending sometime with a calculator i have worked out that we need >> 14,560,000 250x250 pixel tiles to cover GB, with coordinates 0, 0, >> 700000, >> 1300000 at 1m resolution. >> >> This is the configuration file I have been using, and it works and >> creates >> tiles. >> >> >> <?xml version="1.0" encoding="utf-8"?> >> <gwcConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> >> xsi:noNamespaceSchemaLocation="http://geowebcache.org/schema/1.2.0/geowebcache.xsd" >> xmlns="http://geowebcache.org/schema/1.2.0"> >> <version>1.2.0</version> >> <backendTimeout>300</backendTimeout> >> >> <gridSets> >> <!-- >> String name; >> SRS srs; >> BoundingBox extent; >> double[] resolutions; >> double[] scales; >> String[] scaleNames; >> Integer levels; >> Integer tileHeight; >> Integer tileWidth; >> --> >> >> >> >> <gridSet> >> <name>osgb_vml</name> >> <srs><number>27700</number></srs> >> <extent> >> <coords> >> <double>0</double> >> <double>0</double> >> <double>700000</double> >> <double>1300000</double> >> </coords> >> </extent> >> <alignTopLeft>false</alignTopLeft> >> <resolutions> >> <double>1</double> >> </resolutions> >> <tileHeight>250</tileHeight> >> <tileWidth>250</tileWidth> >> </gridSet> >> >> >> </gridSets> >> <layers> >> >> >> <wmsLayer> >> <name>osgb:vml/name> >> <gridSubsets> >> <gridSubset> >> <gridSetName>osgb_vml</gridSetName> >> </gridSubset> >> </gridSubsets> >> >> <wmsUrl><string>http://localhost:8080/geoserver/wms</string></wmsUrl> >> >> <wmsLayers>osgb:areas,osgb:lines,osgb:roadclines,osgb:text</wmsLayers> >> <!-- OPTIONAL The metatiling factors used for this layer >> If not specified, 3x3 metatiling is used for image formats --> >> <metaWidthHeight><int>1</int><int>1</int></metaWidthHeight> >> <bgColor>0xFFFFFF</bgColor> >> </wmsLayer> >> >> </layers> >> </gwcConfiguration> >> >> >> So when i seed this layer and go to the setup page i only the option of >> seeding between zoom level 00 and finish zoom level 00. >> >> Which is fine. >> >> The problem as Pete mentioned above is the shear number of directories >> that >> only have 4 tiles in each folder. >> >> I hope that gives a better picture of what we are up do. I appreciate its >> an >> extreme case but when we get it sorted we will post up the solution and >> the >> timeframes it took. >> >> cheers >> >> Tim >> >> >> >> >> >> Arne Kepp-2 wrote: >> >>> So the plot thickens :) >>> >>> You mentioned in another email that you were just seeding one zoomlevel. >>> So I guess you also made a grid set configuration with very few levels, >>> perhaps just 2? >>> >>> This isn't useful for most GWC users, you need a tile pyramid with many >>> levels sot hat people can gradually zoom in. I am guessing that for >>> OpenSpace you'll be subsampling these tiles to make up intermediate >>> resolutions. >>> >>> So this becomes a very extreme case that the default directory structure >>> was definitely not designed to handle, normally you only have 1 to 16 >>> tiles at zoomlevel 0 , so by the time you get down to 16 million tiles >>> (z=12 or so) they should be spread across 4000 directories with 4000 >>> files each. >>> >>> If you have a thought-out directory structure for your application, I >>> would implement that as a new BlobStore, because your needs will be >>> different from the normal GWC user. Alternatively, you could use the >>> existing tile structure and just define a few resolutions you will not >>> use, so that the level with 16 million tiles is z=12 or so. >>> >>> Note that you need to clear the cache (delete the directory) if you >>> change the grid set definition. >>> >>> -Arne >>> >>> >>> On 06/10/2010 10:05 PM, TMartin wrote: >>> >>>> Hi Arne >>>> >>>> Tim here. Pete and I are both working on the same issue here at OS. >>>> >>>> In each directory there are only 4 tiles of 250x250pixels and 1metre >>>> resolution as specified in our geowebcache configuration xml document. >>>> >>>> If thats the case we may end up with over 3 million directories for the >>>> 14,560,000 tiles we are creating. >>>> >>>> I will try and set it up so you can take a look at if you are >>>> interested? >>>> >>>> Tim >>>> >>>> >>>> >>>> Arne Kepp-2 wrote: >>>> >>>> >>>>> It's a compromise, and this part of the code is definitely made easily >>>>> pluggable so that one can use a different structures for special >>>>> needs. >>>>> I was expecting that we would have a few different implementations by >>>>> now, but apparently not. >>>>> >>>>> Every directory level requires a node to be looked up (move the disk >>>>> head), so too many directories is not good either. Within a directory >>>>> there is a proper index. Listing the content of the cache is generally >>>>> not recommended, I don't know of a single filesystem that is optimized >>>>> for that and most utilities can't handle something of this size. >>>>> "find" >>>>> is usually ok. >>>>> >>>>> But 400 000 directories.... Unless you've discovered a bug, you >>>>> should >>>>> have at least 80 billion tiles? I wouldn't use a conventional >>>>> filesystem for that, and on a fast machine (100 tiles per second) it >>>>> would take 25 years to build a cache of that size. >>>>> >>>>> -Arne >>>>> >>>>> >>>>> On 06/10/2010 11:22 AM, samclemmens wrote: >>>>> >>>>> >>>>>> Cheers, Arne! >>>>>> >>>>>> I'm curious; what was the rationale for creating a directory >>>>>> structure >>>>>> with >>>>>> just one level? We processed a data set earlier this week, which >>>>>> resulted >>>>>> in 400,000 directories. When I tried to an "ls", my PuTTY console >>>>>> froze... >>>>>> >>>>>> Peter >>>>>> >>>>>> >>>>>> >>>>>> Arne Kepp-2 wrote: >>>>>> >>>>>> >>>>>> >>>>>>> 1) "zoom level". For the default EPSG:900913 grid z=0 means the >>>>>>> whole >>>>>>> world in 4 tiles >>>>>>> >>>>>>> 2) Hm,,, good point. I dont think I went back and updated that wiki >>>>>>> page, it was not more than a scratchpad for sharing ideas. The >>>>>>> actual >>>>>>> code is here[1], the difference is that the bit shifting starts with >>>>>>> 2, >>>>>>> effectively making it 2^( 1 + ( z / 2 )). And there's some >>>>>>> zero-padding. >>>>>>> >>>>>>> 3) The catch is that you need to know a little bit of Java, but if >>>>>>> you >>>>>>> do then it's potentially straightforward. It depends what the >>>>>>> OpenSpace >>>>>>> structure is based on, and whether we have similar tokens in GWC. >>>>>>> >>>>>>> blobstore/file contains an implementation of BlobStore. You can copy >>>>>>> the >>>>>>> whole thing to a new package, and then make the FilePathGenerator >>>>>>> output >>>>>>> the structure you want. It is linked into the application using >>>>>>> Spring, >>>>>>> WEB-INF/geowebcache-core-context.xml , if you are working off trunk. >>>>>>> I >>>>>>> would guess that you also want a particular GridSet to go with that, >>>>>>> to >>>>>>> match the resolutions and origin used by OpenSpace. >>>>>>> >>>>>>> It's possible someone has already done it, but so far I have not >>>>>>> heard >>>>>>> about it. >>>>>>> >>>>>>> -Arne >>>>>>> >>>>>>> 1: >>>>>>> http://geowebcache.org/trac/browser/trunk/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/file/FilePathGenerator.java >>>>>>> >>>>>>> >>>>>>> On 06/08/2010 05:33 PM, samclemmens wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm relatively new to GeoWebCache...and I'm a bit puzzled by the >>>>>>>> directory/file name structure that is produced. I have a few >>>>>>>> questions, >>>>>>>> namely: >>>>>>>> >>>>>>>> 1. The documentation states that the naming convention is: >>>>>>>> layername/projection_z/[x/(2^(z/2))]_[y/(2^(z/2))]/x_y.extension >>>>>>>> What >>>>>>>> is >>>>>>>> the z value? >>>>>>>> >>>>>>>> 2. Given the sample output >>>>>>>> .\gwc\roads\EPSG_900913_12\006_019\000848_002540.png, I assume that >>>>>>>> z=12, >>>>>>>> x=848, and y=2540. Obviously the above formula won't produce >>>>>>>> 006_019 >>>>>>>> based >>>>>>>> on these numbers. What conversion is required? >>>>>>>> >>>>>>>> 3. Is it possible to change the directory/file name format? I'd >>>>>>>> like >>>>>>>> to >>>>>>>> create a data set that can be used by Ordnance Survey OpenSpace, >>>>>>>> which >>>>>>>> uses >>>>>>>> a different structure based on the British National Grid. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Peter >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>>>>>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>>>>>> lucky parental unit. See the prize list and enter to win: >>>>>>> http://p.sf.net/sfu/thinkgeek-promo >>>>>>> _______________________________________________ >>>>>>> Geowebcache-users mailing list >>>>>>> Geo...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>>>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>>>> lucky parental unit. See the prize list and enter to win: >>>>> http://p.sf.net/sfu/thinkgeek-promo >>>>> _______________________________________________ >>>>> Geowebcache-users mailing list >>>>> Geo...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------------ >>> ThinkGeek and WIRED's GeekDad team up for the Ultimate >>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the >>> lucky parental unit. See the prize list and enter to win: >>> http://p.sf.net/sfu/thinkgeek-promo >>> _______________________________________________ >>> Geowebcache-users mailing list >>> Geo...@li... >>> https://lists.sourceforge.net/lists/listinfo/geowebcache-users >>> >>> >>> >> > > > ------------------------------------------------------------------------------ > ThinkGeek and WIRED's GeekDad team up for the Ultimate > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the > lucky parental unit. See the prize list and enter to win: > http://p.sf.net/sfu/thinkgeek-promo > _______________________________________________ > Geowebcache-users mailing list > Geo...@li... > https://lists.sourceforge.net/lists/listinfo/geowebcache-users > > -- View this message in context: http://old.nabble.com/GeoWebCache-directory-file-name-structure-tp28819540p28888993.html Sent from the GeoWebCache Users mailing list archive at Nabble.com. |
