Menu
▾
▴
Re: [Jmol-users] Problem w/ electron dens. maps
|
From: Robert H. <ha...@st...> - 2010-02-22 20:49:41
|
OK, progress! Good job, Thomas. Well, it wouldn't be the first time that we have found errors in other systems while working on Jmol. Still proud of finding the bug in ORTEP! Right, so here is where we are: a) Then I have Jmol working correctly now for MRC/CCP4 files, and Eric should be telling us soon that all is to his liking, even with the default of not specifying sigma, since that will go in as sigma=1.0, which is what he wants, I think. b) If you can look into the Uppsala problem, that would be great. Unless I'm way off base there, that needs to be fixed. I just can't see how it can have essentially the same min, max, and mean as the MRC data from the same server and not have those reported in the file. Should be pretty easy for them to check out. c) That said, the CNS/Xplor format very disappointingly is not designed with streaming in mind. One of Jmol's newer features is that it never has to actually load all the volumetric data (the way PyMOL does). It just streams the data in slice by slice and creates the surface on the fly. This is far faster and far more memory efficient and allows Jmol now to display huge surfaces that it never could before. Well, for that to work, you clearly have to have the cutoff initially set. But if it comes at the end of a file -- that's fine for reading a file off the hard drive, but over the web -- where you don't have the equivalent of "file.seek()" to jump to the end first, get the info, and then come back -- that's just not possible. That's too bad. So the point is that while I think it's important that that gets fixed, if it really is in error, fixing that is not going to allow Jmol to read those files efficiently. I hate to think that we would have to read them twice just to do the job. I suppose we could do that.... In any case, the MRC/CCP4 files should be scaling properly now. d) Thomas, if you could look into the electron microscopy business, that would be great as well. Bob On Mon, Feb 22, 2010 at 2:23 PM, Thomas Stout <tho...@gm...>wrote: > > OK, here's what I've found so far: Protein crystallography convention is > to define sigmas as deviations above (and below) the mean. Therefore if a > map has (ideally) a mean of 0.0000, and an RMS of 0.75, then a 1 sigma > contour would be the isosurface at +0.75, a 3sigma map would be drawn at > 2.25 and a -3sigma map at -2.25. If the mean was not at zero for whatever > reason, then 1 sigma is shifted by the mean. Apparently this convention is > so well-accepted that there is not actually anyplace that anyone has been > able to point me to where it is actually written down. I've tried dredging > through the PyMOL code & got lost. I've put in a request for clarification > from Jason Vertrees who is now handling the bulk of the development work > with PyMOL. > > The best way for Jmol to handle all sorts of map imput would probably be to > normalize (and perhaps rescale) when loading the map; then everything can be > handled the same way. I can't speak to electron diffraction conventions -- > I do know someone that works in that field, so I could inquire if you'd > like. > > I am a bit concerned by what you cite for the data coming from the EDS > server at Uppsala. I would expect those values at the end of the CNS/XPLOR > data file to not be mere placeholders. If they are, then that is not useful > (obviously). What EDS is doing, is first calculating the map from structure > factors & writing a DNS6-style binary file, which is then further converted > to any other requested format using "MAPMAN". EDS is doing a great deal of > work to ensure integrity ( > http://eds.bmc.uu.se/eds/eds_help.html#NITTY_GRITTY), so it would be best > to have the format conversion "fixed" if it is not writing the CNS format > correctly. The author of all of that (Gerard Kleywegt; gerard@*ebi*.ac.uk) > is no longer at Uppsala, but is now at the EBI and, I expect, every bit as > interested in getting this right as ever. > > Reference: > GJ Kleywegt, MR Harris, JY Zou, TC Taylor, A Wählby & TA Jones (2004), "*The > Uppsala Electron-Density Server*", Acta Cryst. D60, 2240-2249 > > > On Mon, Feb 22, 2010 at 5:15 AM, Robert Hanson <ha...@st...> wrote: > >> Oh, two more pieces of data: >> >> - In some earlier versions of Jmol I was having it calculate RMSD. Thomas >> pointed out that you can't do that from the data directly. Thus, we must >> rely upon the values given in the data file. >> >> - The Uppsala server also delivers CNS (XPLOR) format data. This data >> (!very inconveniently!) at its END is supposed to have mean and standard >> deviation data. However, my tests do not support that. For example, for >> 1blu.cns, I get: >> >> data min/max/mean = -2.0043898, 4.99725, -0.015510284 >> >> while at the end of the file it just says: >> >> 0.00000E+00 1.00000E+00 >> >> So that's clearly not useful. Basically, they are not including that data, >> and these are just placeholders. The data itself is essentially identical to >> the CCP4 data they serve for the same model, as you can see from the min and >> max there. >> >> I'd be interested to know what PyMOL and these other packages do with that >> CNS data. >> >> Bob >> >> >> >> On Mon, Feb 22, 2010 at 7:01 AM, Robert Hanson <ha...@st...>wrote: >> >>> That would be great, Thomas. Here's what I know: >>> >>> - The MRC/CCP4 files include records that are called dmin, dmax, dmean, >>> and rms. >>> >>> - Jmol tracks min, max, and mean, and those values agree reasonably well >>> with those in the file. For example, for the Uppsala file for 3hyd.ccp4 that >>> Eric has been using, we have: >>> >>> MRC header: dmin,dmax,dmean: -1.1890318,4.998787,-0.014114891 >>> MRC header: rms: 0.74718976 >>> >>> and then from Jmol tracking these values: data min/max/mean = -1.1890318, >>> 4.998787, -0.01760154 >>> >>> I can't explain the slight difference in the mean there, but it's not >>> exactly 0. >>> >>> - Though referred to as "RMS" in the code, the documentation for the MRC >>> format clearly identifies this value as "rms deviation" and not actual >>> "rms". http://ami.scripps.edu/software/mrctools/mrc_specification.php >>> >>> This makes me think that the proper way to present the data is to add in >>> the mean. Otherwise, the mean could be anything, and the cutoff value would >>> be meaningless. >>> >>> - What I implemented in Jmol 11.9.30 is "mean + rmsd*sigma". In previous >>> versions of Jmol I had found that a default of 2 there for "sigma" better >>> fit electron microscopy data. >>> >>> - Jmol needs to be able to represent a broad range of data, including >>> both X-ray and electron microscopy data. Both these fields use the MRC/CCP4 >>> format. So, for example, from EMDB I can get >>> http://emsearch.rutgers.edu/atlas/1003_downloads.html: >>> >>> MRC header: dmin,dmax,dmean: -141.97543,339.987,1.8575257 >>> MRC header: rms: 38.04587 >>> MRC header: labels: CONVERTED FROM SPIDER TO CCP4 USING SPIDER >>> >>> Cutoff set to sigma 1.0 or (mean + rmsDeviation) = 39.903397 >>> >>> Again, the mean isn't 0. >>> >>> I had used 2 sigma because that was the best ad hoc visual fit to what I >>> was seeing in the electron microscopy literature. So I'm wondering now if >>> different fields have different standards. >>> >>> Bob >>> >>> >>> On Mon, Feb 22, 2010 at 1:38 AM, Thomas Stout <tho...@gm...>wrote: >>> >>>> I will try and track down the actual algorithm used - CCP4, O, PyMOL, >>>> MAPMAN, XPLOR/CNS, SOLVE and Phenix (and others) all read and/or write >>>> electron density maps with the same understanding of "sigma" levels, so it >>>> is well standardized in the field. I'm fairly certain that it is just RMSD >>>> but will need to confirm that... If -- in the map that you and Eric are >>>> looking at -- 1 rmsd happens to correspond to the mean, would that account >>>> for the factor of 2 if you are using "mean + rmsd*sigma"? >>>> >>>> -Tom >>>> >>>> Sent from a tiny virtual keyboard. >>>> >>>> On Feb 21, 2010, at 8:56 PM, Robert Hanson <ha...@st...> wrote: >>>> >>>> Eric, it looks like the sigma values associated with Pymol are just 1/2 >>>> the default cutoffs of Jmol. So, for example, the one you say has 0.75 for >>>> the best fit has a default Jmol cutoff of 1.4803, and the one with best fit >>>> 0.33 has a default Jmol cutoff of 0.67. >>>> >>>> Maybe Thomas can explain how sigma is related in this way. What we can >>>> do is create a new sigma parameter for the Jmol isosurface command that will >>>> match Pymol's sigma -- at least for (some) ccp4 files. Worth a shot. >>>> >>>> Bob >>>> >>>> >>>> On Sun, Feb 21, 2010 at 10:09 PM, Robert Hanson < <ha...@st...> >>>> ha...@st...> wrote: >>>> >>>>> That sounds about right. >>>>> >>>>> We need to track down the definition of "sigma" that pymol is using. >>>>> Jmol's definition of "cutoff" is straightforward -- just the actual >>>>> isovalue. Not having any familiarity at all with python, I haven't been able >>>>> to find this in the Pymol code. >>>>> >>>>> Perhaps someone else can spot that.... >>>>> >>>>> Bob >>>>> >>>>> >>>>> >>>>> On Sun, Feb 21, 2010 at 7:59 AM, Eric Martz < <mar...@ya...> >>>>> mar...@ya...> wrote: >>>>> >>>>>> OK, I misinterpreted the scaling differences in my previous report. >>>>>> The problem appears to be a simple scaling factor difference between Jmol's >>>>>> isomesh cutoff value and PyMOL's isomesh sigma value. (Contrary to my >>>>>> previous interpretation, I don't think the scaling varies from the center of >>>>>> the molecule.) >>>>>> >>>>>> However, I now notice that the scale factor (cutoff vs. sigma ratio >>>>>> that makes the two isomeshes the same) varies with the size of the molecule >>>>>> loaded. >>>>>> >>>>>> Detailed demonstrations are at the top link on this list: >>>>>> <http://www.umass.edu/molvis/tests/> >>>>>> http://www.umass.edu/molvis/tests/ >>>>>> >>>>>> -Eric >>>>>> >>>>>> --- On Fri, 2/19/10, Robert Hanson < <ha...@st...> >>>>>> ha...@st...> wrote: >>>>>> >>>>>> > From: Robert Hanson < <ha...@st...>ha...@st...> >>>>>> > Subject: Re: [Jmol-users] Problem w/ electron dens. maps >>>>>> > To: <jmo...@li...> >>>>>> jmo...@li... >>>>>> > Date: Friday, February 19, 2010, 7:40 PM >>>>>> > Exactly the problem. We have wildly >>>>>> > different data sets -- from electron diffraction to X-ray >>>>>> > data. Where did we end up with that discussion about sigma? >>>>>> > I can certainly look at the Pymol code and see what Warren >>>>>> > did there. >>>>>> > >>>>>> > >>>>>> > Bob >>>>>> > >>>>>> > On Fri, Feb 19, 2010 at 11:37 AM, >>>>>> > Thomas Stout < <tho...@gm...>tho...@gm...> >>>>>> > wrote: >>>>>> > >>>>>> > >>>>>> > But electron density maps can be on arbitrary scales, so >>>>>> > the value "1.0" may very well be meaningless >>>>>> > (except in the particular case where normalization has made >>>>>> > 1.0=1e-/A**3), and will certainly produce wildly different >>>>>> > results from dataset to dataset depending on the provenance >>>>>> > of each particular dataset (scale factors, etc). The >>>>>> > "standard" in macromolecular crystallography is to >>>>>> > view density maps at 1.0sigma & "difference >>>>>> > maps" at 3.0sigma. >>>>>> > >>>>>> > >>>>>> > >>>>>> > -Tom >>>>>> > >>>>>> > >>>>>> > On >>>>>> > Fri, Feb 19, 2010 at 7:49 AM, Robert Hanson < <ha...@st...> >>>>>> ha...@st...> >>>>>> > wrote: >>>>>> > >>>>>> > >>>>>> > Eric, I think this is just that Pymol has a different >>>>>> > definition -- 1.0 sigma is not 1.0 value. Jmol uses actual >>>>>> > value 1.0, not "sigma" 1.0. >>>>>> > >>>>>> > Bob >>>>>> > >>>>>> > >>>>>> > On Fri, Feb 19, 2010 at 7:26 AM, Eric Martz < <mar...@ya...> >>>>>> mar...@ya...> >>>>>> > wrote: >>>>>> > >>>>>> > Dear Bob, >>>>>> > >>>>>> > >>>>>> > >>>>>> > I'm sorry to report that Jmol 11.9.29 is not displaying >>>>>> > ccp4 electron density maps correctly as isomeshes. I have >>>>>> > demonstrated a major problem here (hopefully not major to >>>>>> > fix): >>>>>> > >>>>>> > >>>>>> > >>>>>> > <http://www.umass.edu/molvis/tests/jmol_edm_test5/> >>>>>> http://www.umass.edu/molvis/tests/jmol_edm_test5/ >>>>>> > >>>>>> > >>>>>> > >>>>>> > -Eric >>>>>> > >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Download Intel® Parallel Studio Eval >>>>>> Try the new software tools for yourself. Speed compiling, find bugs >>>>>> proactively, and fine-tune applications for parallel performance. >>>>>> See why Intel Parallel Studio got high marks during beta. >>>>>> <http://p.sf.net/sfu/intel-sw-dev>http://p.sf.net/sfu/intel-sw-dev >>>>>> _______________________________________________ >>>>>> Jmol-users mailing list >>>>>> <Jmo...@li...>Jmo...@li... >>>>>> <https://lists.sourceforge.net/lists/listinfo/jmol-users> >>>>>> https://lists.sourceforge.net/lists/listinfo/jmol-users >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Robert M. Hanson >>>>> Professor of Chemistry >>>>> St. Olaf College >>>>> 1520 St. Olaf Ave. >>>>> Northfield, MN 55057 >>>>> <http://www.stolaf.edu/people/hansonr> >>>>> http://www.stolaf.edu/people/hansonr >>>>> phone: 507-786-3107 >>>>> >>>>> >>>>> If nature does not answer first what we want, >>>>> it is better to take what answer we get. >>>>> >>>>> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 >>>>> >>>> >>>> >>>> >>>> -- >>>> Robert M. Hanson >>>> Professor of Chemistry >>>> St. Olaf College >>>> 1520 St. Olaf Ave. >>>> Northfield, MN 55057 >>>> <http://www.stolaf.edu/people/hansonr> >>>> http://www.stolaf.edu/people/hansonr >>>> phone: 507-786-3107 >>>> >>>> >>>> If nature does not answer first what we want, >>>> it is better to take what answer we get. >>>> >>>> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Download Intel® Parallel Studio Eval >>>> >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> >>>> _______________________________________________ >>>> Jmol-users mailing list >>>> Jmo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/jmol-users >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Download Intel® Parallel Studio Eval >>>> Try the new software tools for yourself. Speed compiling, find bugs >>>> proactively, and fine-tune applications for parallel performance. >>>> See why Intel Parallel Studio got high marks during beta. >>>> http://p.sf.net/sfu/intel-sw-dev >>>> _______________________________________________ >>>> Jmol-users mailing list >>>> Jmo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/jmol-users >>>> >>>> >>> >>> >>> -- >>> Robert M. Hanson >>> Professor of Chemistry >>> St. Olaf College >>> 1520 St. Olaf Ave. >>> Northfield, MN 55057 >>> http://www.stolaf.edu/people/hansonr >>> phone: 507-786-3107 >>> >>> >>> If nature does not answer first what we want, >>> it is better to take what answer we get. >>> >>> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 >>> >> >> >> >> -- >> Robert M. Hanson >> Professor of Chemistry >> St. Olaf College >> 1520 St. Olaf Ave. >> Northfield, MN 55057 >> http://www.stolaf.edu/people/hansonr >> phone: 507-786-3107 >> >> >> If nature does not answer first what we want, >> it is better to take what answer we get. >> >> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> Jmol-users mailing list >> Jmo...@li... >> https://lists.sourceforge.net/lists/listinfo/jmol-users >> >> > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Jmol-users mailing list > Jmo...@li... > https://lists.sourceforge.net/lists/listinfo/jmol-users > > -- Robert M. Hanson Professor of Chemistry St. Olaf College 1520 St. Olaf Ave. Northfield, MN 55057 http://www.stolaf.edu/people/hansonr phone: 507-786-3107 If nature does not answer first what we want, it is better to take what answer we get. -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 |