Menu
▾
▴
gnuplot-beta
|
From: Justace C. <pro...@co...> - 2004-03-03 04:27:09
|
So,
I have this project where I have computed a bunch of channel
correlations. The entire matrix is like 1800x1800 with one side of the
diaganal set to NaN since it is symetric. Now to the root of the
problem. I use the pm3d portion of gnuplot to create a 2D map of the
matrix. Here is the snipplit of code that I use...
-----------------------SNIP
if($save) {
print GNUPLOT "
set terminal postscript eps color palfuncparam 500
set output \"${saveFilename}\"
set size 3/10,3/7
";
} else {
GNUPLOT->autoflush(1);
}
print GNUPLOT "
set xrange [1:1792]
set yrange [1:1792]
set cbrange [-1:1]
# unset colorbox
unset ytics
set xtics (\"129\" 129, \"257\" 257, \"385\" 385, \"513\" 513,
\"641\" 641, \"0\" 769, \"129\" 897, \"257\" 1025, \"385\" 1153, \"513\"
1281, \"641\" 1409, \"769\" 1537, \"897\" 1665)
set y2tics (\"129\" 129, \"257\" 257, \"385\" 385, \"513\" 513,
\"641\" 641, \"0\" 769, \"129\" 897, \"257\" 1025, \"385\" 1153, \"513\"
1281, \"641\" 1409, \"769\" 1537, \"897\" 1665)
set format y \"%4.0f\"
set format x \"%4.0f\"
set size ratio 1
set title \"${titleClean}\"
set y2label \"Channel\"
set xlabel \"Channel\"
set pm3d map
set palette defined (0 \"white\", 1 \"black\", 2 \"blue\")
# set palette defined (-1 \"red\", -0.5 \"black\", 0.5 \"black\", 1
\"blue\")
splot '${tmpFilename}' matrix title \"\"
";
if(! $save) {
print GNUPLOT "
pause -1;
";
getc();
}
close(GNUPLOT);
--------------------------END SNIP
I used the first set size command to try to bring the file size down, of
course it did not work. The resultant file is 120Meg. This is crazy.
Am I doing somehting wrong?
Justace
|
|
From: Ethan A M. <merritt@u.washington.edu> - 2004-03-03 05:56:31
|
On Tuesday 02 March 2004 08:20 pm, Justace Clutter wrote: > > I have this project where I have computed a bunch of channel > correlations. The entire matrix is like 1800x1800 with one side of the > diaganal set to NaN since it is symetric. Now to the root of the > problem. I use the pm3d portion of gnuplot to create a 2D map of the > matrix. Here is the snipplit of code that I use... > set terminal postscript eps color palfuncparam 500 This is a case where PostScript is probably the wrong format to choose for output. The PostScript output file will contain a separate command for every point (or box or line or...) plotted, even if they all fall on top of each other. For very large datasets, where there are more data points than the resolution of your output device, you are better off using a pixel-based output format like PNG. That way the output size is fixed in advance. No matter how many data points map to a single pixel, it's still only one pixel. -- Ethan A Merritt Department of Biochemistry & Biomolecular Structure Center University of Washington, Seattle |
|
From: Petr M. <mi...@ph...> - 2004-03-03 07:01:49
|
> I have this project where I have computed a bunch of channel > correlations. The entire matrix is like 1800x1800 with one side of the > diaganal set to NaN since it is symetric. Now to the root of the > problem. I use the pm3d portion of gnuplot to create a 2D map of the > matrix. Here is the snipplit of code that I use... > > I used the first set size command to try to bring the file size down, of > course it did not work. The resultant file is 120Meg. This is crazy. Ethan has explained why the ps file does not depend on "set size". Decrease size of pm3d map in ps file can be done by: 1. Postprocessing your ps file -- see scripts pm3dCompress.awk and pm3dConvertToImage.awk in gnuplot sources or in http://gnuplot.sourceforge.net/scripts/ 2. Use "with pixels style" patch by Daniel at the "Patches" section on sourceforge. --- Petr Mikulik |
|
From: Arnd B. <arn...@we...> - 2004-03-03 07:21:34
|
On Wed, 3 Mar 2004, Petr Mikulik wrote: [...] > 2. Use "with pixels style" patch by Daniel at the "Patches" section on > sourceforge. And in that case I think using RLE instead of the present scheme should reduce the file size by another relevant factor. (However, though this seems pretty straight forward, someone has to code this and at the moment I am completely busy with other stuff ;-) Best, Arnd |
|
From: Daniel J S. <dan...@ie...> - 2004-03-03 16:06:46
|
Arnd Baecker wrote: >On Wed, 3 Mar 2004, Petr Mikulik wrote: > >[...] > > > >>2. Use "with pixels style" patch by Daniel at the "Patches" section on >>sourceforge. >> >> > >And in that case I think using RLE instead of the present scheme >should reduce the file size by another relevant factor. >(However, though this seems pretty straight forward, >someone has to code this and at the moment I am completely >busy with other stuff ;-) > Well, run length encoding is great for applications like fax data and the sort, i.e., a single bit image depth. That's because there are typically long stretches of white space where the run length really achieves compression. However, I'm not sure what RLE will get you in color or gray scale images. My experience with angiography images and the sort is that maybe 50% lossless compression is the best on average. There are forms of lossy compression, but this takes consideration. If someone on the list has experience with compact representation of various image types, that would be helpful... I feel that coding is half the problem. The other half is simply making a decision what to do, i.e., agreeing on what the format or syntax should be. It can in fact be a fun little project to implement some of these things. I just went to the library and looked at the Adobe PostScript Standard book and picked the most flexible beginning format, which to me seemed to be ASCII85. The reason is that it would be lossless compression (keeping everyone fairly happy from the start), various PostScript viewers would likely be able to read it, and there would be no question of whether we should use, say, and MPEG compression library routine. That is, I didn't see any sense in going full tilt into writing, say, an MPEG routine if it wasn't certain it would get used or if the developers may decided using some existing library would be the smartest maintenance alternative. Dan |
|
From: Daniel J S. <dan...@ie...> - 2004-03-03 16:34:23
|
Daniel J Sebald wrote: > My experience with angiography images and the sort is that maybe 50% > lossless compression is the best on average. (That's not RLE encoding, but something else.) Dan |
|
From: Daniel J S. <dan...@ie...> - 2004-03-03 16:31:51
|
Justace Clutter wrote: >I like the idea of this pixels based image. The fact that the image >could be 10 megs does not bother me. > Well, in my case my web space is only 10 M. > It is the idea that an image could >be 120 megs bothers me. So I got the patch and applyed. It did not go >clean as one hunk did not make it. But I fixed it by hand. I was >patching against 3.8k.1. So, I guess that I do not understand the >syntax. Here is the datafile that I have, or really a sample: > >0.999849 0.652204 0.678995 0.658985 0.585981 0.566624 0.547193 0.615902 >0.539023 0.556559 0.547578 0.53746 0.522146 0.535582 0.530735 0.531924 >0.554801 0.518362 0.479106 0.527362 0.534836 0.546124 0.522834 0.508155 >0.54706 0.469279 0.581292 0.468283 0.535739 0.557979 0.490128 0.536971 >0.530735 0.506848 0.523189 0.531732 0.550917 0.484002 0.57789 0.460027 >0.556111 0.535427 0.508878 0.558746 0.459494 0.578337 0.45805 0.595234 > >So I have a grid here, 8x6, that I would like displayed. Each of these >floats would need to be mapped onto a color. PM3D did this great. So >when I try to use the matrix modifier with just plot it is not happy, >then I tried to use splot, still not happy. Here is what I was trying.. > >gnuplot> plot 'tmp.data' matrix with image > >GNUPLOT (plot_image): Image grid must be at least 2 x 2. > >Warning: empty x range [0.999849:0.999849], adjusting to >[0.989851:1.00985] > >GNUPLOT (plot_image): Image grid must be at least 2 x 2. > >gnuplot> splot 'tmp.data' matrix with image > >GNUPLOT (plot_image): Image grid must be at least 2 x 2. > >gnuplot> set view map >gnuplot> splot 'tmp.data' matrix with image > >GNUPLOT (plot_image): Image grid must be at least 2 x 2. > >gnuplot> > >Can you give me some ideas on how I can use the image style with this >setup? I looked at the information on the image but it seems pretty >biased toward images like the tux and not array'd data. Thanks in >advance. > Well, here is the issue. You are putting the data in ASCII format in what looks to you like a grid. However, Gnuplot hasn't been set up to read ASCII data in such a way. It is set up to read ordered tuples of data, such as x1 y1 z1 x2 y2 z2 ... where the position is part of the data specification. Telling Gnuplot what the underlying grid is without specifying it is the problem. That is why the "binary" file specifications have extra qualifiers to indicate grid dimensions, grid spacing, etc. Another alternative for binary is "Gnuplot binary" where the first line and row of the matrix give the x and y spacing information. I've generally not used that for images, but I think it should work. (Type "help splot binary" at Gnuplot's command line for more info on that format.) The reason you may want to avoid putting your image in ASCII format is that you are playing the numbers game again. Your example above has 9 characters for each pixel element. 1800 x 1800 x 9 or roughly 27 M at a minimum, and if when you add the (x,y) coordinates it goes to 81 M. That's too big. If in the program or software you are using to generate the data you could multiply by 2^16 and then save the values in binary as unsigned 16 bit ints you would be better off. Can you do that? Dan >Justace > >On Wed, 2004-03-03 at 01:00, Daniel J Sebald wrote: > > >>Justace Clutter wrote: >> >> >> >>>So, >>> >>> I have this project where I have computed a bunch of channel >>>correlations. The entire matrix is like 1800x1800 with one side of the >>>diaganal set to NaN since it is symetric. Now to the root of the >>>problem. I use the pm3d portion of gnuplot to create a 2D map of the >>>matrix. Here is the snipplit of code that I use... >>> >>> >>> >><snip> >> >> >> >>>I used the first set size command to try to bring the file size down, of >>>course it did not work. The resultant file is 120Meg. This is crazy. >>>Am I doing somehting wrong? >>> >>> >>> >>Justace, >> >>Am I understanding correctly that you are essentially attempting to >>display an 1800x1800 matrix as an image? Ethan's comment is correct >>that each element, i.e., pixel, will be drawn as a rectangle, i.e., >>polygon. There is a lot of ASCII text associated with each rectangle. >> You might try the image patch on SourceForge (something originally >>titled "with pixels"). That will put the image into a PostScript file >>in a compact form. >> >>Realize though that 1800x1800 is a lot of data. Saved as bytes, that >>alone would be roughly 3M. I've tried 1000x1000 images with no >>problems, but things really do slow down. So, unless it is necessary to >>keep such high resolution for the image, your first step might be to >>down sample the data appropriately. Keep in mind what your eventual >>display will be. (For example, people often put high res photos on >>their web pages which take close to a minute to download. However, the >>display is usually low resolution so a condensed image would have served >>equally well and, in fact, would save space on their server.) Ethan's >>suggestion of PNG is a good idea because that inherently brings down the >>resolution. >> >>Dan >> >> |
|
From: Daniel J S. <dan...@ie...> - 2004-03-03 19:37:28
|
Justace Clutter wrote: >I have been using this ASCII grid format for quite a while now. That is >the definition from the help pages for the matrix datafile format. Am I >wrong with this? > Oh yeah, I forgot about "matrix". The issue there is that "matrix" is designed for 3d data, i.e., splot. (List members, if anything I say here is incorrect, please correct me.) There is a routine in "datafile.c" called df_3dmatrix. I think it only get's called from "plot3d.c". There is no way to call it from "plot2d.c". Inside "datafile.c" where the word "matrix" is tested for there seems to be no test for whether or not the mode is "splot" or "plot". In other words, if I'm not mistaken, Gnuplot should not allow "matrix" to be used in "plot" mode. In the documentation for "matrix" there is reference to a mesh, so I assume there is an "splot" restriction. Actually, in the image patch, I modified things so that a mode variable is passed into that routine so it is possible to identify whether "splot" or "plot" was issued. I probably needed that information for something else, but it could be used to rule out "matrix" when issued with "plot". Or, for that matter plot2d.c could be modified to allow data files in "matrix" format. To you Justace, this is all probably neither here nor there. But it gets back to a point I've tried to make in previous discussions, that 2d and 3d formats have diverged in some ways. I think the "datafile.c" and related routines could be cleaned up a great deal, focusing more on just getting data from a file into Gnuplot and not so much on what the eventually plotting mode will look like. Suffice it to say that "matrix" is currently being ignored in "plot" mode. Sorry. I suppose it would be possible to hack a quick fix together in the image patch (I'm not sure how much work it would take), but I'd prefer to not do that. Instead, I think it would be better for the group to discuss how best to clean up the datafile routines post 4.0, then make the change. I say that because there is the alternative of binary data, and I think it is a better habit to work with binary data for large files. Do developers here agree that, for large files, binary data in "plot" mode is preferable to ascii matrix? Dan >Justace > >On Wed, 2004-03-03 at 10:46, Daniel J Sebald wrote: > > >>Justace Clutter wrote: >> >> >> >>>I like the idea of this pixels based image. The fact that the image >>>could be 10 megs does not bother me. >>> >>> >>> >>Well, in my case my web space is only 10 M. >> >> >> >>> It is the idea that an image could >>>be 120 megs bothers me. So I got the patch and applyed. It did not go >>>clean as one hunk did not make it. But I fixed it by hand. I was >>>patching against 3.8k.1. So, I guess that I do not understand the >>>syntax. Here is the datafile that I have, or really a sample: >>> >>>0.999849 0.652204 0.678995 0.658985 0.585981 0.566624 0.547193 0.615902 >>>0.539023 0.556559 0.547578 0.53746 0.522146 0.535582 0.530735 0.531924 >>>0.554801 0.518362 0.479106 0.527362 0.534836 0.546124 0.522834 0.508155 >>>0.54706 0.469279 0.581292 0.468283 0.535739 0.557979 0.490128 0.536971 >>>0.530735 0.506848 0.523189 0.531732 0.550917 0.484002 0.57789 0.460027 >>>0.556111 0.535427 0.508878 0.558746 0.459494 0.578337 0.45805 0.595234 >>> >>>So I have a grid here, 8x6, that I would like displayed. Each of these >>>floats would need to be mapped onto a color. PM3D did this great. So >>>when I try to use the matrix modifier with just plot it is not happy, >>>then I tried to use splot, still not happy. Here is what I was trying.. >>> >>>gnuplot> plot 'tmp.data' matrix with image >>> >>>GNUPLOT (plot_image): Image grid must be at least 2 x 2. >>> >>>Warning: empty x range [0.999849:0.999849], adjusting to >>>[0.989851:1.00985] >>> >>>GNUPLOT (plot_image): Image grid must be at least 2 x 2. >>> >>>gnuplot> splot 'tmp.data' matrix with image >>> >>>GNUPLOT (plot_image): Image grid must be at least 2 x 2. >>> >>>gnuplot> set view map >>>gnuplot> splot 'tmp.data' matrix with image >>> >>>GNUPLOT (plot_image): Image grid must be at least 2 x 2. >>> >>>gnuplot> >>> >>>Can you give me some ideas on how I can use the image style with this >>>setup? I looked at the information on the image but it seems pretty >>>biased toward images like the tux and not array'd data. Thanks in >>>advance. >>> >>> >>> >>Well, here is the issue. You are putting the data in ASCII format in >>what looks to you like a grid. However, Gnuplot hasn't been set up to >>read ASCII data in such a way. It is set up to read ordered tuples of >>data, such as >> >>x1 y1 z1 >>x2 y2 z2 >>... >> >>where the position is part of the data specification. >> >>Telling Gnuplot what the underlying grid is without specifying it is the >>problem. That is why the "binary" file specifications have extra >>qualifiers to indicate grid dimensions, grid spacing, etc. >> >>Another alternative for binary is "Gnuplot binary" where the first line >>and row of the matrix give the x and y spacing information. I've >>generally not used that for images, but I think it should work. (Type >>"help splot binary" at Gnuplot's command line for more info on that format.) >> >>The reason you may want to avoid putting your image in ASCII format is >>that you are playing the numbers game again. Your example above has 9 >>characters for each pixel element. 1800 x 1800 x 9 or roughly 27 M at a >>minimum, and if when you add the (x,y) coordinates it goes to 81 M. >> That's too big. If in the program or software you are using to >>generate the data you could multiply by 2^16 and then save the values in >>binary as unsigned 16 bit ints you would be better off. Can you do that? >> >>Dan >> >> >> >>>Justace >>> >>>On Wed, 2004-03-03 at 01:00, Daniel J Sebald wrote: >>> >>> >>> >>> >>>>Justace Clutter wrote: >>>> >>>> >>>> >>>> >>>> >>>>>So, >>>>> >>>>> I have this project where I have computed a bunch of channel >>>>>correlations. The entire matrix is like 1800x1800 with one side of the >>>>>diaganal set to NaN since it is symetric. Now to the root of the >>>>>problem. I use the pm3d portion of gnuplot to create a 2D map of the >>>>>matrix. Here is the snipplit of code that I use... >>>>> >>>>> >>>>> >>>>> >>>>> >>>><snip> >>>> >>>> >>>> >>>> >>>> >>>>>I used the first set size command to try to bring the file size down, of >>>>>course it did not work. The resultant file is 120Meg. This is crazy. >>>>>Am I doing somehting wrong? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>Justace, >>>> >>>>Am I understanding correctly that you are essentially attempting to >>>>display an 1800x1800 matrix as an image? Ethan's comment is correct >>>>that each element, i.e., pixel, will be drawn as a rectangle, i.e., >>>>polygon. There is a lot of ASCII text associated with each rectangle. >>>>You might try the image patch on SourceForge (something originally >>>>titled "with pixels"). That will put the image into a PostScript file >>>>in a compact form. >>>> >>>>Realize though that 1800x1800 is a lot of data. Saved as bytes, that >>>>alone would be roughly 3M. I've tried 1000x1000 images with no >>>>problems, but things really do slow down. So, unless it is necessary to >>>>keep such high resolution for the image, your first step might be to >>>>down sample the data appropriately. Keep in mind what your eventual >>>>display will be. (For example, people often put high res photos on >>>>their web pages which take close to a minute to download. However, the >>>>display is usually low resolution so a condensed image would have served >>>>equally well and, in fact, would save space on their server.) Ethan's >>>>suggestion of PNG is a good idea because that inherently brings down the >>>>resolution. >>>> >>>>Dan >>>> >>>> >>>> >>>> >> >> >>------------------------------------------------------- >>SF.Net is sponsored by: Speed Start Your Linux Apps Now. >>Build and deploy apps & Web services for Linux with >>a free DVD software kit from IBM. Click Now! >>http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click >>_______________________________________________ >>gnuplot-beta mailing list >>gnu...@li... >>https://lists.sourceforge.net/lists/listinfo/gnuplot-beta >> >> > > > > > -- Dan Sebald email: daniel DOT sebald AT ieee DOT org URL: http://acer-access DOT com/~dsebald AT acer-access DOT com/ |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X