From: Adrian C. <ac...@gm...> - 2008-05-30 10:35:58
|
Hey Christian, Being far from SVN these days, I end up being the last to update so I only just downloaded all the .tif files you recently committed to the SVN. While others have commented on this, perhaps this has not gotten back to you yet. You should *not* be committing data to the SVN without good reason. We have all made this mistake and recently we just spent a bunch of time cleaning up the SVN after ourselves so you are not the first. I am surprised you did not read about this in the Developer's Guide---when Confluence comes back up I'll go look and make sure this admonition is there. Also it's too bad your mentor didn't look over your commits before you made them to make sure they were clean---please talk to your mentor when you are about to do anything unusual. Obviously, you need data for testing. Please look at the sample-data module to see if you can reuse what is there. If you can not, then we should add the minimal amount of data to that module so you can work. You should decide what you need and send an email to the list asking everyone if this makes sense. Note that if you need to have massively large images, to cut up, mosaic or otherwise manipulate, you should generate those on the fly from smaller fragments for your test. For now, you should remove the files from the SVN so we don't each have them in all our working copies. The elegant way to do this would be to back out all your work to the last commit prior to adding the files. Then you can re-commit all that work other than the images. You'll have to learn how to commit reverse patches: ask your mentor and use the svn book as your friend. Doing this well will allow us easily to clean up the repository itself. thanks, adrian |
From: Daniele R. <dan...@gm...> - 2008-05-30 10:43:23
|
Hi Adrian, Before you sent this email, I was writing an email to Christian saying to avoid committing data in SVN. Then I avoid "brute" delete of these data leaving Christian to follow your procedure. Cheers, Daniele On Fri, May 30, 2008 at 12:35 PM, Adrian Custer <ac...@gm...> wrote: > Hey Christian, > > Being far from SVN these days, I end up being the last to update so I > only just downloaded all the .tif files you recently committed to the > SVN. While others have commented on this, perhaps this has not gotten > back to you yet. > > > You should *not* be committing data to the SVN without good reason. > > We have all made this mistake and recently we just spent a bunch of time > cleaning up the SVN after ourselves so you are not the first. I am > surprised you did not read about this in the Developer's Guide---when > Confluence comes back up I'll go look and make sure this admonition is > there. Also it's too bad your mentor didn't look over your commits > before you made them to make sure they were clean---please talk to your > mentor when you are about to do anything unusual. > > Obviously, you need data for testing. Please look at the sample-data > module to see if you can reuse what is there. If you can not, then we > should add the minimal amount of data to that module so you can work. > You should decide what you need and send an email to the list asking > everyone if this makes sense. Note that if you need to have massively > large images, to cut up, mosaic or otherwise manipulate, you should > generate those on the fly from smaller fragments for your test. > > For now, you should remove the files from the SVN so we don't each have > them in all our working copies. The elegant way to do this would be to > back out all your work to the last commit prior to adding the files. > Then you can re-commit all that work other than the images. You'll have > to learn how to commit reverse patches: ask your mentor and use the svn > book as your friend. Doing this well will allow us easily to clean up > the repository itself. > > thanks, > adrian > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Geotools-devel mailing list > Geo...@li... > https://lists.sourceforge.net/lists/listinfo/geotools-devel > -- ------------------------------------------------------- Eng. Daniele Romagnoli Software Engineer GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy phone: +39 0584983027 fax: +39 0584983027 mob: +39 328 0559267 http://www.geo-solutions.it ------------------------------------------------------- |
From: Adrian C. <ac...@gm...> - 2008-05-30 11:09:02
|
On Fri, 2008-05-30 at 12:43 +0200, Daniele Romagnoli wrote: > Hi Adrian, > Before you sent this email, I was writing an email to Christian saying > to avoid committing data in SVN. > > Then I avoid "brute" delete of these data leaving Christian to follow > your procedure. Please don't "leave" him to do it, if you are acting as mentor. It's non-trivial and I'm sure he could use some confirmation that he's doing things correctly. thanks, adrian |
From: Daniele R. <dan...@gm...> - 2008-05-30 11:16:27
|
On Fri, May 30, 2008 at 1:09 PM, Adrian Custer <ac...@gm...> wrote: > On Fri, 2008-05-30 at 12:43 +0200, Daniele Romagnoli wrote: > > Hi Adrian, > > Before you sent this email, I was writing an email to Christian saying > > to avoid committing data in SVN. > > > > Then I avoid "brute" delete of these data leaving Christian to follow > > your procedure. > > Please don't "leave" him to do it, if you are acting as mentor. It's > non-trivial and I'm sure he could use some confirmation that he's doing > things correctly. Ops. My fault, bad italian to english translation... Maybe I should use "let" or something else... anyway without "abandoning" him to this task :). Cheers, Daniele > > > thanks, > adrian > > -- ------------------------------------------------------- Eng. Daniele Romagnoli Software Engineer GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy phone: +39 0584983027 fax: +39 0584983027 mob: +39 328 0559267 http://www.geo-solutions.it ------------------------------------------------------- |
From: Christian M. <chr...@nv...> - 2008-05-30 12:49:15
|
Some questions: 1) Should I rename gt-imagemosaicJDBC to gt-imagemosiac-jdbc ? 2) For testing I decided to use 3 pyramids, the third pyramid contains only 1 image. The biggest dir contains the tiles for the original image, using 9.6 MB. The whole src dir needs 16 MB. I found some modules which have equal size, e.g widgets-swing-pending needs about 15 MB, or the ogc module. The tiles for itself will not change. I know the binary file problem from CVS assuming there are the same problems with SVN. Since mosaicing and selecting the correct pyramid is the key feature, I need these images for testing. The question for me is: Removing the pictures means no tests, no good idea. Removing the biggest level (in MB) will reduce the size to 6-7 MB, then I have 2 pyramids and have to adjust the tests. (not too much work) Sorry I read the developers guide, but this problem was not obvious to me. christian Daniele Romagnoli writes: > On Fri, May 30, 2008 at 1:09 PM, Adrian Custer <ac...@gm...> wrote: > >> On Fri, 2008-05-30 at 12:43 +0200, Daniele Romagnoli wrote: >> > Hi Adrian, >> > Before you sent this email, I was writing an email to Christian saying >> > to avoid committing data in SVN. >> > >> > Then I avoid "brute" delete of these data leaving Christian to follow >> > your procedure. >> >> Please don't "leave" him to do it, if you are acting as mentor. It's >> non-trivial and I'm sure he could use some confirmation that he's doing >> things correctly. > > > Ops. My fault, > bad italian to english translation... Maybe I should use "let" or something > else... anyway without "abandoning" him to this task :). > > Cheers, > Daniele > > > > >> >> >> thanks, >> adrian >> >> > > > -- > ------------------------------------------------------- > Eng. Daniele Romagnoli > Software Engineer > > GeoSolutions S.A.S. > Via Carignoni 51 > 55041 Camaiore (LU) > Italy > > phone: +39 0584983027 > fax: +39 0584983027 > mob: +39 328 0559267 > > > http://www.geo-solutions.it > > ------------------------------------------------------- |
From: Andrea A. <aa...@op...> - 2008-05-30 13:04:17
|
Christian Müller ha scritto: > Some questions: > > 1) > Should I rename gt-imagemosaicJDBC to gt-imagemosiac-jdbc ? This would be consistent with the other modules, so it's a good idea. > 2) For testing I decided to use 3 pyramids, the third pyramid contains only > 1 image. The biggest dir contains the tiles for the original image, using > 9.6 MB. The whole src dir needs 16 MB. I found some modules which have > equal size, e.g widgets-swing-pending needs about 15 MB, or the ogc module. Wow, this is bad. How can widgets-swing-pending need that much data? Are you sure you're not including in your size count stuff that is generated during the build? Please avoid that, you don't need that much data. You can make a pyramid we a few hundreds of kilobytes, size does not matter for unit testing, and for performance testing, the current 12MB are not enough anyways. The ideal (but this is more work, so nobody is asking you to do so) would be to generate your test data on the fly. I mean, you don't really need a specific image content, so you can just generate buffered images on the fly, draw a number or something inside of them, and save them on the disk. > The tiles for itself will not change. I know the binary file problem from > CVS assuming there are the same problems with SVN. Since mosaicing and > selecting the correct pyramid is the key feature, I need these images for > testing. The problem is not about how svn handles svn data, but put yourself in the pants of someone that has to check out geotools from a country with slow internet connection. Moreover, I believe there are performance issues in svn when the repository becomes too big. Cheers Andrea |
From: Daniele R. <dan...@gm...> - 2008-05-30 13:53:48
|
On Fri, May 30, 2008 at 3:04 PM, Andrea Aime <aa...@op...> wrote: > Christian Müller ha scritto: > >> Some questions: >> 1) >> Should I rename gt-imagemosaicJDBC to gt-imagemosiac-jdbc ? >> > > This would be consistent with the other modules, so it's a good idea. A small trivial note to avoid misunderstandings: we are talking about the directory name within the physical svn repo. (As suggested here: http://docs.codehaus.org/display/GEOT/5.2+Naming+Conventions) Since actually it is imagemosaicJDBC you can rename it as imagemosaic-jdbc. (Note that there aren't directories starting with "gt-"). > > > 2) For testing I decided to use 3 pyramids, the third pyramid contains >> only 1 image. The biggest dir contains the tiles for the original image, >> using 9.6 MB. The whole src dir needs 16 MB. I found some modules which >> have equal size, e.g widgets-swing-pending needs about 15 MB, or the ogc >> module. >> > > Wow, this is bad. How can widgets-swing-pending need that much data? > Are you sure you're not including in your size count stuff that is > generated during the build? > > Please avoid that, you don't need that much data. You can make a pyramid > we a few hundreds of kilobytes, size does not matter for unit testing, > and for performance testing, the current 12MB are not enough anyways. > > The ideal (but this is more work, so nobody is asking you to do so) would > be to generate your test data on the fly. I mean, you don't really > need a specific image content, so you can just generate buffered images on > the fly, draw a number or something inside of them, and save them on the > disk. > > The tiles for itself will not change. I know the binary file problem from >> CVS assuming there are the same problems with SVN. Since mosaicing and >> selecting the correct pyramid is the key feature, I need these images for >> testing. >> > > The problem is not about how svn handles svn data, but put yourself > in the pants of someone that has to check out geotools from a country > with slow internet connection. Moreover, I believe there are performance > issues in svn when the repository becomes too big. > > Cheers > Andrea > Cheers, Daniele -- ------------------------------------------------------- Eng. Daniele Romagnoli Software Engineer GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy phone: +39 0584983027 fax: +39 0584983027 mob: +39 328 0559267 http://www.geo-solutions.it ------------------------------------------------------- |
From: Christian M. <chr...@nv...> - 2008-05-30 14:19:11
|
I refactored the tests and save 8 MB, now using 7.8 MB. The changes have been commited. Renaming will be done next week since I am out of time now Cheers christian Daniele Romagnoli writes: > On Fri, May 30, 2008 at 3:04 PM, Andrea Aime <aa...@op...> wrote: > >> Christian Müller ha scritto: >> >>> Some questions: >>> 1) >>> Should I rename gt-imagemosaicJDBC to gt-imagemosiac-jdbc ? >>> >> >> This would be consistent with the other modules, so it's a good idea. > > > A small trivial note to avoid misunderstandings: we are talking about the > directory name within the physical svn repo. (As suggested here: > http://docs.codehaus.org/display/GEOT/5.2+Naming+Conventions) > > Since actually it is imagemosaicJDBC you can rename it as imagemosaic-jdbc. > (Note that there aren't directories starting with "gt-"). > > >> >> >> 2) For testing I decided to use 3 pyramids, the third pyramid contains >>> only 1 image. The biggest dir contains the tiles for the original image, >>> using 9.6 MB. The whole src dir needs 16 MB. I found some modules which >>> have equal size, e.g widgets-swing-pending needs about 15 MB, or the ogc >>> module. >>> >> >> Wow, this is bad. How can widgets-swing-pending need that much data? >> Are you sure you're not including in your size count stuff that is >> generated during the build? >> >> Please avoid that, you don't need that much data. You can make a pyramid >> we a few hundreds of kilobytes, size does not matter for unit testing, >> and for performance testing, the current 12MB are not enough anyways. >> >> The ideal (but this is more work, so nobody is asking you to do so) would >> be to generate your test data on the fly. I mean, you don't really >> need a specific image content, so you can just generate buffered images on >> the fly, draw a number or something inside of them, and save them on the >> disk. >> >> The tiles for itself will not change. I know the binary file problem from >>> CVS assuming there are the same problems with SVN. Since mosaicing and >>> selecting the correct pyramid is the key feature, I need these images for >>> testing. >>> >> >> The problem is not about how svn handles svn data, but put yourself >> in the pants of someone that has to check out geotools from a country >> with slow internet connection. Moreover, I believe there are performance >> issues in svn when the repository becomes too big. >> >> Cheers >> Andrea >> > > Cheers, > Daniele > > -- > ------------------------------------------------------- > Eng. Daniele Romagnoli > Software Engineer > > GeoSolutions S.A.S. > Via Carignoni 51 > 55041 Camaiore (LU) > Italy > > phone: +39 0584983027 > fax: +39 0584983027 > mob: +39 328 0559267 > > > http://www.geo-solutions.it > > ------------------------------------------------------- |
From: Daniele R. <dan...@gm...> - 2008-05-30 14:39:18
|
Hi Christian, I guess this approach doesn't solve the original problem. (test data is too heavy in size). Now a question for the GeoTools people: should a single 250K file be "size-acceptable"? I'm asking if having a single 250K zip file to be extracted during tests, could make sense. Please, before committing other stuff let's find a good solution. Cheers, Daniele On Fri, May 30, 2008 at 4:06 PM, Christian Müller <chr...@nv...> wrote: > I refactored the tests and save 8 MB, now using 7.8 MB. > The changes have been commited. > > Renaming will be done next week since I am out of time now > > Cheers > christian > > > Daniele Romagnoli writes: > > > On Fri, May 30, 2008 at 3:04 PM, Andrea Aime <aa...@op...> > wrote: > > > >> Christian Müller ha scritto: > >> > >>> Some questions: > >>> 1) > >>> Should I rename gt-imagemosaicJDBC to gt-imagemosiac-jdbc ? > >>> > >> > >> This would be consistent with the other modules, so it's a good idea. > > > > > > A small trivial note to avoid misunderstandings: we are talking about the > > directory name within the physical svn repo. (As suggested here: > > http://docs.codehaus.org/display/GEOT/5.2+Naming+Conventions) > > > > Since actually it is imagemosaicJDBC you can rename it as > imagemosaic-jdbc. > > (Note that there aren't directories starting with "gt-"). > > > > > >> > >> > >> 2) For testing I decided to use 3 pyramids, the third pyramid contains > >>> only 1 image. The biggest dir contains the tiles for the original > image, > >>> using 9.6 MB. The whole src dir needs 16 MB. I found some modules > which > >>> have equal size, e.g widgets-swing-pending needs about 15 MB, or the > ogc > >>> module. > >>> > >> > >> Wow, this is bad. How can widgets-swing-pending need that much data? > >> Are you sure you're not including in your size count stuff that is > >> generated during the build? > >> > >> Please avoid that, you don't need that much data. You can make a pyramid > >> we a few hundreds of kilobytes, size does not matter for unit testing, > >> and for performance testing, the current 12MB are not enough anyways. > >> > >> The ideal (but this is more work, so nobody is asking you to do so) > would > >> be to generate your test data on the fly. I mean, you don't really > >> need a specific image content, so you can just generate buffered images > on > >> the fly, draw a number or something inside of them, and save them on the > >> disk. > >> > >> The tiles for itself will not change. I know the binary file problem > from > >>> CVS assuming there are the same problems with SVN. Since mosaicing and > >>> selecting the correct pyramid is the key feature, I need these images > for > >>> testing. > >>> > >> > >> The problem is not about how svn handles svn data, but put yourself > >> in the pants of someone that has to check out geotools from a country > >> with slow internet connection. Moreover, I believe there are performance > >> issues in svn when the repository becomes too big. > >> > >> Cheers > >> Andrea > >> > > > > Cheers, > > Daniele > > > > -- > > ------------------------------------------------------- > > Eng. Daniele Romagnoli > > Software Engineer > > > > GeoSolutions S.A.S. > > Via Carignoni 51 > > 55041 Camaiore (LU) > > Italy > > > > phone: +39 0584983027 > > fax: +39 0584983027 > > mob: +39 328 0559267 > > > > > > http://www.geo-solutions.it > > > > ------------------------------------------------------- > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Geotools-devel mailing list > Geo...@li... > https://lists.sourceforge.net/lists/listinfo/geotools-devel > > -- ------------------------------------------------------- Eng. Daniele Romagnoli Software Engineer GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy phone: +39 0584983027 fax: +39 0584983027 mob: +39 328 0559267 http://www.geo-solutions.it ------------------------------------------------------- |
From: Adrian C. <ac...@gm...> - 2008-05-30 14:52:22
|
On Fri, 2008-05-30 at 16:39 +0200, Daniele Romagnoli wrote: > Hi Christian, > I guess this approach doesn't solve the original problem. (test data > is too heavy in size). > > Now a question for the GeoTools people: > should a single 250K file be "size-acceptable"? I'm asking if having a > single 250K zip file to be extracted during tests, could make sense. Yes, that sounds perfectly reasonable. However... 1) It should live in sample-data, *not* in a module (so it can be shared) 2) Haven't you all been doing coverage work? Where's the data you all are using? You should all have data in sample-data so can't we use that? --adrian |
From: Simone G. <sim...@gm...> - 2008-05-30 15:24:53
|
I understand you care particularly about this topic since you have spent a lot of time cleaning up the svn. To answer your (multiple as usual) questions: 1> the sample data is partly split between modules (old commits) and partly in the sample-data module (new commits). 2> Leaving aside the initial two questions, since I do not have time for jokes, sorry, I agree that it would be much better to put all the test data in the same directory, but I think that such a decision should be made at the PMC level and then should be reflected inside the developers guide and finally enforced on the svn commit level. At that time we should start a discussion about sample data size and sample data placement for the various modules, because frankly starting this discussion randomly like you have done is quite useless. Conclusion is, you have my vote if you want to propose more strict rules for managing sample data and for requiring developers/committer to respect them; you won't have any useful answear if you approach the problem on a random per-module basis. Ciao, Simone. On Fri, May 30, 2008 at 4:52 PM, Adrian Custer <ac...@gm...> wrote: > On Fri, 2008-05-30 at 16:39 +0200, Daniele Romagnoli wrote: >> Hi Christian, >> I guess this approach doesn't solve the original problem. (test data >> is too heavy in size). >> >> Now a question for the GeoTools people: >> should a single 250K file be "size-acceptable"? I'm asking if having a >> single 250K zip file to be extracted during tests, could make sense. > > Yes, that sounds perfectly reasonable. However... > > 1) It should live in sample-data, *not* in a module (so it can be > shared) > > 2) Haven't you all been doing coverage work? Where's the data you all > are using? You should all have data in sample-data so can't we use that? > > --adrian > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Geotools-devel mailing list > Geo...@li... > https://lists.sourceforge.net/lists/listinfo/geotools-devel > -- ------------------------------------------------------- Eng. Simone Giannecchini President /CEO GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy phone: +39 0584983027 fax: +39 0584983027 mob: +39 333 8128928 http://www.geo-solutions.it ------------------------------------------------------- |
From: Jody G. <jga...@re...> - 2008-05-30 21:14:10
|
Adrian Custer wrote: > Yes, that sounds perfectly reasonable. However... > > 1) It should live in sample-data, *not* in a module (so it can be shared) > We have the following page for test data: - http://docs.codehaus.org/display/GEOT/5.+8+Test+Data While it was the stated aim of the sample-data module to hold shared test data; I don't think it was written up as project policy. The instructions for use of the TestData class and test-data folders are however important; with out following these instructions we get some crazy cross platform build failures... > 2) Haven't you all been doing coverage work? Where's the data you all are using? You should all have data in sample-data so can't we use that? > The page here is a bit scant on details: - http://docs.codehaus.org/display/GEOTOOLS/Sample+Data+Module Jody |
From: Adrian C. <ac...@gm...> - 2008-05-30 13:56:32
|
On Fri, 2008-05-30 at 14:37 +0200, Christian Müller wrote: > Some questions: Please, work with your mentor on this. If that person doesn't have answers for you, then come on back to the list. > 2) For testing I decided to use 3 pyramids, the third pyramid contains only > 1 image. The biggest dir contains the tiles for the original image, using > 9.6 MB. The whole src dir needs 16 MB. I found some modules which have > equal size, e.g widgets-swing-pending needs about 15 MB, or the ogc module. Yes, indeed. So you should be surprised if, starting out, you are among the biggest modules. (BTW, you need to exclude the .svn data from your count. eratosthenes:.../widgets-swing-pending> du -sh 13M . eratosthenes:.../widgets-swing-pending> du -sh --exclude="\.svn" 3.4M . ) So please look for a strategy with your mentors. From what I understood they have been doing image work so probably already have some answer to your issue. Then ask again here if you are still stuck after that. good luck, --adrian |
From: Adrian C. <ac...@gm...> - 2008-05-30 14:05:33
|
Christian, PLEASE STOP COMMITTING. Weird stuff is happening. You need to operate carefully right now. So please, go slowly, explain to someone what you are planning to do, ask if it's going to work, get the go ahead and only then, do it. svn up just gave me: ... D modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.fix A modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.fix D modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.shp A modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.shp D modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.dbf A modules/unsupported/imagemosaicJDBC/src/test/resources/0/index.dbf D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_1.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_1.tif D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_2_1.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_2_1.tif D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_2.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_2.tif D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_3_1.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_3_1.tif D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_2_2.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_2_2.tif D modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_3.tif A modules/unsupported/imagemosaicJDBC/src/test/resources/0/oek4000_1_3.tif ... If you are using an IDE, stop and use the commandline until you figure out what is going on. --adrian |