Menu
▾
▴
pyphant-devel — Discussions about the inner workings and future developments.
You can subscribe to this list here.
| 2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(13) |
Nov
(50) |
Dec
(40) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2008 |
Jan
(49) |
Feb
(72) |
Mar
(18) |
Apr
(27) |
May
(40) |
Jun
(52) |
Jul
(26) |
Aug
(8) |
Sep
(12) |
Oct
(26) |
Nov
(13) |
Dec
(14) |
| 2009 |
Jan
(13) |
Feb
(7) |
Mar
(8) |
Apr
(11) |
May
(4) |
Jun
(2) |
Jul
(7) |
Aug
(1) |
Sep
(3) |
Oct
|
Nov
(1) |
Dec
(1) |
| 2010 |
Jan
(1) |
Feb
(8) |
Mar
(3) |
Apr
(9) |
May
(16) |
Jun
(8) |
Jul
(7) |
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(6) |
Dec
(12) |
|
From: Klaus Z. <kla...@fm...> - 2010-05-02 12:17:30
|
Hi all, Rolf wrote: > Andreas wrote: > > But on the other hand aggregation of FMF-files contained within > > a ZIP-archive is an important feature, which should be formally > > introduced in FMF 1.2. But this feature does need SampleContainers. > I don't get the point on the issue with the zip-files. > [...] > From my simple point of view I would see three handlings/issues: > > You get a bunch of fmf-files which where stored in one zip-file > to save bandwidth (and maybe help handling) > > You get a bunch of files stored in an zip file and some of them > where fmf-files. > > An third approach would be to store the data compressed while > keeping the meta-information human-readable > > IMHO the first two issues would be part of the application > and not the standard. The third one would be something which > my be interesting, but we should be aware that it breaks some > point of the standard: the human readability of data. Just to clarify, here is an example where we use zip files right now: One of our customers maps a physical sample with a small light beam, getting a spectrum for every point on an arbitrary grid. Each such spectrum is saved in a fmf file and the set of all files making up the sample are zipped together. Then on loading *of the zip file* the fmf parser aggregates the data from all the fmf files, using some heuristics to generate a sample container with the necessary redundancy mostly removed. For example, if a key value pair in a general section appears identically in all files it will end up as an attribute of the sample container, whereas it will result in a column if it changes across the fmf files. This means of course, that if some parameter is identical across all files by accident, it will not produce the correct column, but a surprising attribute. This happened to us for example when we extracted a certain row of pixels from the sample. In that case all files had the same y coordinate which messed up our pyphant recipe for the expected why column did no longer exist in the respective sample containers. This is what Andreas had in mind, when he wrote about zip files: There should be some specification allowing for general attributes and so on. So what do you think: 1) Do we need a specification for aggregate data? 2) Is zip a good means for that? 3) Can we get proper inspiration from prior art? What about jar files, eggs? Best, Klaus |
|
From: Rolf W. <ro...@di...> - 2010-04-30 19:40:17
|
On Thu, 29 Apr 2010 07:34:09 +0200, "Dr. Andreas W. Liehr"
<ob...@fm...> wrote:
> Hallo *,
>
Hello *,
>> Klaus hesitates a little bit yesterday about "forking fmf" from
>> pyphant (setting up an own sf project for fmf) - maybe some more
>> discussion about the pros and cons is needed.
> The idea is to create a lightweight Python package, which could easily
> be integrated e.g. in scipy. The difficult question is, where to draw
> the line?
Maybe my last mail lends to misinterpretions (sorry, Klaus) - Klaus
doesn't
hesitate to fork the fmf from pyphant (to entangle them) he was more on
hesitating to set up an own project on SF for this.
However after some discussion today we uncertain things and will prepare
the jump to sf soon.
>
> There is no doubt, that the quantities module should belong to the FMF
> package. But, what about DataContainers? Pyphant as such, does not play
> any restrictions on the data objects to be transfered between the
> workers, but on the other hand uses DataContainers heavily for
> Visualisation and the Knowledge Cloud.
>
> Of course a FMF module could return a dictionary of dictionary
> containing the metadata and a set of numpy arrays, containing the data
> columns. But on the other hand aggregation of FMF-files contained within
> a ZIP-archive is an important feature, which should be formally
> introduced in FMF 1.2. But this feature does need SampleContainers.
Klaus also pronounced the very good idea of using an sax-like approach
where
we generate events wo help people parsing the files. I would suggest to
give people both possibilities: "In Memory" (dict - type) and "event
driven"
(sax type). Btw: my idea behind the fork of fmf was to give other the
possibility to use the fmf without having to install the whole pyphant.
And also to give some libs in other languages - so people from other
software-
project can use it easily.
I don't get the point on the issue with the zip-files.
>From my simple point of few I would see three handlings/issues:
You get a bunch of fmf-files which where stored in one zip-file
to save bandwidth (and maybe help handling)
You get a bunch of files stored in an zip file and some of them
where fmf-files.
An third approach would be to store the data compressed while
keeping the meta-information human-readable
IMHO the first two issues would be part of the application
and not the standard. The third one would be something which
my be interesting, but we should be aware that it breaks some
point of the standard: the human readability of data.
>
> All the best,
> Andreas
Kind regs,
Rolf
--
Security is an illusion - Datasecurity twice
Rolf Würdemann - ro...@di...
GnuPG fingerprint: 7383 348F 67D1 CD27 C90F DDD0 86A3 31B6 67F0
D02F
jabber: ro...@di... ECF127C7 EAB85F87 BC75ACB5 2EC646D4 9211A31
|
|
From: Dr. A. W. L. <ob...@fm...> - 2010-04-30 03:51:29
|
Hallo *, > Klaus hesitates a little bit yesterday about "forking fmf" from > pyphant (setting up an own sf project for fmf) - maybe some more > discussion about the pros and cons is needed. The idea is to create a lightweight Python package, which could easily be integrated e.g. in scipy. The difficult question is, where to draw the line? There is no doubt, that the quantities module should belong to the FMF package. But, what about DataContainers? Pyphant as such, does not play any restrictions on the data objects to be transfered between the workers, but on the other hand uses DataContainers heavily for Visualisation and the Knowledge Cloud. Of course a FMF module could return a dictionary of dictionary containing the metadata and a set of numpy arrays, containing the data columns. But on the other hand aggregation of FMF-files contained within a ZIP-archive is an important feature, which should be formally introduced in FMF 1.2. But this feature does need SampleContainers. All the best, Andreas |
|
From: Rolf W. <ro...@di...> - 2010-04-28 09:41:41
|
On Tue, 27 Apr 2010 05:50:16 +0200, "Dr. Andreas W. Liehr" <ob...@fm...> wrote: > Dear *, > Hello again, >> Even in this discussions the suggestion of setting up an own project on >> SF >> for the FMF arises - which I would second. > I totally agree. Klaus hesitates a little bit yesterday about "forking fmf" from pyphant (setting up an own sf project for fmf) - maybe some more discussion about the pros and cons is needed. > >> Question a) >> >> who should do this > Any volunteers are welcome and I'd also like to participate as much as > my exhausted spare time allows for. If we've clarified the way I have no problems setting up the sf project, but don't want to bumper in somebody's garden. > >> Question b) >> >> I would keep in mind that there should be support for other languages >> as python - should this get reflected in the repository structure? > Very good idea. Are there any suggestions from other projects? > I've strolled 'round the network yesterday. As it seems there are not many other projects which wents the way of doing native implementations. Only netcdf gives native libs - and this seems not to be an open source project - maybe an look on freshmeat will help. Most libs which supports more than one language are working with bindings - which is easy to understood if you look of the work for the individual lib (fftw for example) against the work of providing bindings. If you look at file formats implemented in different languages (csv, ini, png for example) these are usually own projects which are not interconnected to each other (only by specification) >> >> Also with the issue of support for other languages there arises an >> suggestion of setting up distinct levels of compatibility. See: >> https://dev.fmf.uni-freiburg.de/projects/pyphant1/wiki/Introducing_levels_of_compatibility >> >> What's you opinion on this? > If we are heading for supporting different languages/software packages a > level of compatibility is clearly needed. At present the FMF realizes > this level of compatibility intrinsically, because a key:value pair of > strings is accepted in any way, which corresponds to level 0. ACK > > Sofware levels: > Level 0: Is realized by libraries supporting INI format. Only the > reading of data columns in [*data]-sections has to be added. I agree - even if we should do code reuse and not library reuse (which means we shouldn't read the file twice ;) > > Level 1: Could be realized as a mapping of [*data definitions]-sections > to [*data] columns. Yes, even if we should do some basic checking of the variables (also in the [*reference] part of the system (if I understand Klaus right) > > Level 2: E.g. like the reference implementation in Pyphant. ACK > > In general I am certain, that everybody does need units, but maybe > different ones. Therefore we should allow for a [*unit > definitions]-section. I second this (Bees/Beestack) > > All the best, > Andreas -- Security is an illusion - Datasecurity twice Rolf Würdemann - ro...@di... GnuPG fingerprint: 7383 348F 67D1 CD27 C90F DDD0 86A3 31B6 67F0 D02F jabber: ro...@di... ECF127C7 EAB85F87 BC75ACB5 2EC646D4 9211A31 |
|
From: Dr. A. W. L. <ob...@fm...> - 2010-04-27 04:07:22
|
Dear *, > Even in this discussions the suggestion of setting up an own project on SF > for the FMF arises - which I would second. I totally agree. > Question a) > > who should do this Any volunteers are welcome and I'd also like to participate as much as my exhausted spare time allows for. > Question b) > > I would keep in mind that there should be support for other languages > as python - should this get reflected in the repository structure? Very good idea. Are there any suggestions from other projects? > > Also with the issue of support for other languages there arises an > suggestion of setting up distinct levels of compatibility. See: > https://dev.fmf.uni-freiburg.de/projects/pyphant1/wiki/Introducing_levels_of_compatibility > > What's you opinion on this? If we are heading for supporting different languages/software packages a level of compatibility is clearly needed. At present the FMF realizes this level of compatibility intrinsically, because a key:value pair of strings is accepted in any way, which corresponds to level 0. Sofware levels: Level 0: Is realized by libraries supporting INI format. Only the reading of data columns in [*data]-sections has to be added. Level 1: Could be realized as a mapping of [*data definitions]-sections to [*data] columns. Level 2: E.g. like the reference implementation in Pyphant. In general I am certain, that everybody does need units, but maybe different ones. Therefore we should allow for a [*unit definitions]-section. All the best, Andreas |
|
From: <ro...@bo...> - 2010-04-19 20:03:59
|
Hi everybody, as discussed with Klaus and Andreas some days ago, it would make sense to fork the Full-Metadata-Format (FMF) from pyphant. Even if they belong to each other, the FMF is an format for data-communication and archivation as the pyphant is used for the distinct part of processing data. Even in this discussions the suggestion of setting up an own project on SF for the FMF arises - which I would second. Question a) who should do this Question b) I would keep in mind that there should be support for other languages as python - should this get reflected in the repository structure? Also with the issue of support for other languages there arises an suggestion of setting up distinct levels of compatibility. See: https://dev.fmf.uni-freiburg.de/projects/pyphant1/wiki/Introducing_levels_of_compatibility What's you opinion on this? Kind reg's, Rolf |
|
From: <zk...@us...> - 2010-04-19 18:15:16
|
Revision: 677
http://pyphant.svn.sourceforge.net/pyphant/?rev=677&view=rev
Author: zklaus
Date: 2010-04-19 18:15:08 +0000 (Mon, 19 Apr 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Removed chaco from visualizers
Modified Paths:
--------------
trunk/src/pyphant/setup.py
trunk/src/workers/ImageProcessing/setup.py
trunk/src/workers/OSC/setup.py
trunk/src/workers/Statistics/setup.py
trunk/src/workers/fmfile/setup.py
trunk/src/workers/tools/setup.py
Removed Paths:
-------------
trunk/src/visualizers/chaco/chaco/ImageVisualizer.py
trunk/src/visualizers/chaco/chaco/__init__.py
trunk/src/visualizers/chaco/setup.cfg
trunk/src/visualizers/chaco/setup.py
Modified: trunk/src/pyphant/setup.py
===================================================================
--- trunk/src/pyphant/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/pyphant/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
__id__ = '$Id: $'
Deleted: trunk/src/visualizers/chaco/chaco/ImageVisualizer.py
===================================================================
--- trunk/src/visualizers/chaco/chaco/ImageVisualizer.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/visualizers/chaco/chaco/ImageVisualizer.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,102 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright (c) 2006-2007, Rectorate of the University of Freiburg
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution.
-# * Neither the name of the Freiburg Materials Research Center,
-# University of Freiburg nor the names of its contributors may be used to
-# endorse or promote products derived from this software without specific
-# prior written permission.
-#
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
-# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
-# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
-# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
-# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
-# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-u"""
-"""
-
-__id__ = "$Id$"
-__author__ = "$Author$"
-__version__ = "$Revision$"
-# $Source$
-
-import scipy
-import wx
-from enthought.chaco2 import api as chaco2
-from enthought.chaco2.tools import api as tools
-from enthought.enable2.wx_backend import Window
-from pyphant.core.Connectors import TYPE_IMAGE
-from enthought.chaco2.default_colormaps \
- import color_map_name_dict
-
-class PlotFrame(wx.Frame):
- name = 'New Image Visualizer'
- def __init__(self, fieldContainer, *args, **kw):
- kw["size"] = (600,600)
- wx.Frame.__init__(*(self,None,)+args, **kw)
- data = chaco2.ArrayPlotData()
- data.set_data('imagedata', fieldContainer.data.data)
- plot = chaco2.Plot(data)
- plot.img_plot('imagedata',
- xbounds=fieldContainer.dimensions[0].data,
- ybounds=fieldContainer.dimensions[1].data,
- colormap = color_map_name_dict['jet'])
- plot.overlays.append(tools.SimpleZoom(plot, tool_mode="box", always_on=True))
- self.plot_window = Window(parent=self, component=plot)
- sizer=wx.BoxSizer(wx.HORIZONTAL)
- sizer.Add(self.plot_window.control, 1, wx.EXPAND)
- self.SetSizer(sizer)
- self.SetAutoLayout(True)
- self.Show(True)
- return
-
-## class ImageVisualizer(object):
-## name='Image Visualizer'
-## def __init__(self, fieldContainer):
-## self.fieldContainer = fieldContainer
-## self.execute()
-
-## def execute(self):
-## pylab.ioff()
-## self.figure = pylab.figure()
-## xmin=scipy.amin(self.fieldContainer.dimensions[0].data)
-## xmax=scipy.amax(self.fieldContainer.dimensions[0].data)
-## ymin=scipy.amin(self.fieldContainer.dimensions[1].data)
-## ymax=scipy.amax(self.fieldContainer.dimensions[1].data)
-
-## pylab.imshow(self.fieldContainer.data, extent=(xmin, xmax, ymin, ymax))
-## pylab.xlabel(self.fieldContainer.dimensions[0].label)
-## pylab.ylabel(self.fieldContainer.dimensions[1].label)
-## pylab.title(self.fieldContainer.longname)
-
-## class F(pylab.Formatter):
-## def __init__(self, container, *args, **kwargs):
-## self.container=container
-## def __call__(self, x, pos=None):
-## try:
-## return str(x*self.container.unit).replace('mu',r'\textmu{}')
-## except IndexError, error:
-## return str(x)
-## ax=pylab.gca()
-## pylab.colorbar(format=F(self.fieldContainer))
-## pylab.ion()
-## pylab.show()
-
-
Deleted: trunk/src/visualizers/chaco/chaco/__init__.py
===================================================================
--- trunk/src/visualizers/chaco/chaco/__init__.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/visualizers/chaco/chaco/__init__.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,39 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright (c) 2006-2007, Rectorate of the University of Freiburg
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution.
-# * Neither the name of the Freiburg Materials Research Center,
-# University of Freiburg nor the names of its contributors may be used to
-# endorse or promote products derived from this software without specific
-# prior written permission.
-#
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
-# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
-# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
-# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
-# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
-# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-u"""
-"""
-
-__id__ = "$Id$"
-__author__ = "$Author$"
-__version__ = "$Revision$"
-# $Source$
-
Deleted: trunk/src/visualizers/chaco/setup.cfg
===================================================================
--- trunk/src/visualizers/chaco/setup.cfg 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/visualizers/chaco/setup.cfg 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +0,0 @@
-# -*- coding: utf-8 -*-
-[egg_info]
-tag_build = .dev
-tag_svn_revision = 1
Deleted: trunk/src/visualizers/chaco/setup.py
===================================================================
--- trunk/src/visualizers/chaco/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/visualizers/chaco/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,32 +0,0 @@
-#!/usr/bin/env python2.5
-# -*- coding: utf-8 -*-
-
-u"""
-Pyphant Chaco Visualizers
-This package contains visualizers that make use of the enthought.chaco
-package.
-"""
-
-__author__ = "Klaus Zimmermann, Andreas W. Liehr"
-
-__id__ = '$Id$'
-__revision__ = '$Revision: 25 $'
-
-VERSION = '0.1'
-
-from setuptools import setup, find_packages
-
-setup(
- name = "Pyphant Chaco Visualizer",
- version = VERSION,
- author = __author__,
- description = __doc__,
- install_requires=['pyphant>=0.4alpha3',
- 'enthought.chaco2'],
- packages = find_packages(),
- entry_points = """
- [pyphant.visualizers]
- pil.image = chaco.ImageVisualizer:PlotFrame
- """
- )
-
Modified: trunk/src/workers/ImageProcessing/setup.py
===================================================================
--- trunk/src/workers/ImageProcessing/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/workers/ImageProcessing/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Modified: trunk/src/workers/OSC/setup.py
===================================================================
--- trunk/src/workers/OSC/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/workers/OSC/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Modified: trunk/src/workers/Statistics/setup.py
===================================================================
--- trunk/src/workers/Statistics/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/workers/Statistics/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Modified: trunk/src/workers/fmfile/setup.py
===================================================================
--- trunk/src/workers/fmfile/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/workers/fmfile/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Modified: trunk/src/workers/tools/setup.py
===================================================================
--- trunk/src/workers/tools/setup.py 2010-04-19 18:06:36 UTC (rev 676)
+++ trunk/src/workers/tools/setup.py 2010-04-19 18:15:08 UTC (rev 677)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.5
+#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-04-19 18:06:42
|
Revision: 676
http://pyphant.svn.sourceforge.net/pyphant/?rev=676&view=rev
Author: zklaus
Date: 2010-04-19 18:06:36 +0000 (Mon, 19 Apr 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Update of test runner.
Modified Paths:
--------------
trunk/src/support/xmlrunner.py
Modified: trunk/src/support/xmlrunner.py
===================================================================
--- trunk/src/support/xmlrunner.py 2010-04-12 09:49:12 UTC (rev 675)
+++ trunk/src/support/xmlrunner.py 2010-04-19 18:06:36 UTC (rev 676)
@@ -3,20 +3,24 @@
"""
# Written by Sebastian Rittau <sr...@jr...> and placed in
-# the Public Domain. With contributions by Paolo Borelli.
+# the Public Domain. With contributions by Paolo Borelli and others.
-__revision__ = "$Id: /private/python/stdlib/xmlrunner.py 16654 2007-11-12T12:46:35.368945Z srittau $"
+from __future__ import with_statement
+__version__ = "0.1"
+
import os.path
import re
import sys
import time
import traceback
import unittest
-from StringIO import StringIO
from xml.sax.saxutils import escape
-from StringIO import StringIO
+try:
+ from StringIO import StringIO
+except ImportError:
+ from io import StringIO
class _TestInfo(object):
@@ -63,9 +67,9 @@
"method": self._method,
"time": self._time,
})
- if self._failure != None:
+ if self._failure is not None:
self._print_error(stream, 'failure', self._failure)
- if self._error != None:
+ if self._error is not None:
self._print_error(stream, 'error', self._error)
stream.write('</testcase>\n')
@@ -74,7 +78,7 @@
text = escape(str(error[1]))
stream.write('\n')
stream.write(' <%s type="%s">%s\n' \
- % (tagname, escape(str(error[0])), text))
+ % (tagname, _clsname(error[0]), text))
tb_stream = StringIO()
traceback.print_tb(error[2], None, tb_stream)
stream.write(escape(tb_stream.getvalue()))
@@ -82,6 +86,10 @@
stream.write(' ')
+def _clsname(cls):
+ return cls.__module__ + "." + cls.__name__
+
+
class _XMLTestResult(unittest.TestResult):
"""A test result class that stores result as XML.
@@ -176,13 +184,7 @@
result = _XMLTestResult(classname)
start_time = time.time()
- # TODO: Python 2.5: Use the with statement
- old_stdout = sys.stdout
- old_stderr = sys.stderr
- sys.stdout = StringIO()
- sys.stderr = StringIO()
-
- try:
+ with _fake_std_streams():
test(result)
try:
out_s = sys.stdout.getvalue()
@@ -192,13 +194,10 @@
err_s = sys.stderr.getvalue()
except AttributeError:
err_s = ""
- finally:
- sys.stdout = old_stdout
- sys.stderr = old_stderr
time_taken = time.time() - start_time
result.print_report(stream, time_taken, out_s, err_s)
- if self._stream == None:
+ if self._stream is None:
stream.close()
return result
@@ -213,7 +212,21 @@
stream.""")
+class _fake_std_streams(object):
+
+ def __enter__(self):
+ self._orig_stdout = sys.stdout
+ self._orig_stderr = sys.stderr
+ sys.stdout = StringIO()
+ sys.stderr = StringIO()
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ sys.stdout = self._orig_stdout
+ sys.stderr = self._orig_stderr
+
+
class XMLTestRunnerTest(unittest.TestCase):
+
def setUp(self):
self._stream = StringIO()
@@ -221,7 +234,7 @@
"""Run the test suite against the supplied test class and compare the
XML result against the expected XML string. Fail if the expected
- string doesn't match the actual string. All time attribute in the
+ string doesn't match the actual string. All time attributes in the
expected string should have the value "0.000". All error and failure
messages are reduced to "Foobar".
@@ -238,6 +251,8 @@
# string.
got = re.sub(r'(?s)<failure (.*?)>.*?</failure>', r'<failure \1>Foobar</failure>', got)
got = re.sub(r'(?s)<error (.*?)>.*?</error>', r'<error \1>Foobar</error>', got)
+ # And finally Python 3 compatibility.
+ got = got.replace('type="builtins.', 'type="exceptions.')
self.assertEqual(expected, got)
@@ -310,7 +325,7 @@
"""
class TestTest(unittest.TestCase):
def test_foo(self):
- print "Test"
+ sys.stdout.write("Test\n")
self._try_test_run(TestTest, """<testsuite errors="0" failures="0" name="unittest.TestSuite" tests="1" time="0.000">
<testcase classname="__main__.TestTest" name="test_foo" time="0.000"></testcase>
<system-out><![CDATA[Test
@@ -326,7 +341,7 @@
"""
class TestTest(unittest.TestCase):
def test_foo(self):
- print >>sys.stderr, "Test"
+ sys.stderr.write("Test\n")
self._try_test_run(TestTest, """<testsuite errors="0" failures="0" name="unittest.TestSuite" tests="1" time="0.000">
<testcase classname="__main__.TestTest" name="test_foo" time="0.000"></testcase>
<system-out><![CDATA[]]></system-out>
@@ -365,14 +380,5 @@
runner.run(unittest.makeSuite(TestTest))
-class XMLTestProgram(unittest.TestProgram):
- def runTests(self):
- if self.testRunner is None:
- self.testRunner = XMLTestRunner()
- unittest.TestProgram.runTests(self)
-
-main = XMLTestProgram
-
-
if __name__ == "__main__":
- main(module=None)
+ unittest.main()
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-04-12 09:49:19
|
Revision: 675
http://pyphant.svn.sourceforge.net/pyphant/?rev=675&view=rev
Author: zklaus
Date: 2010-04-12 09:49:12 +0000 (Mon, 12 Apr 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Moved test h5 again. Maybe final destination?
Modified Paths:
--------------
trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py
Modified: trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py 2010-04-06 10:09:55 UTC (rev 674)
+++ trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py 2010-04-12 09:49:12 UTC (rev 675)
@@ -76,8 +76,8 @@
os.remove(h5name)
def testGetHTTPFile(self):
- host = "omnibus.uni-freiburg.de"
- remote_dir = "/~s8klzimm"
+ host = "pyphant.sourceforge.net"
+ remote_dir = ""
url = "http://" + host + remote_dir + "/knowledgemanager-http-test.h5"
# Get remote file and load DataContainer
filename, headers = urllib.urlretrieve(url)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-04-06 10:10:03
|
Revision: 674
http://pyphant.svn.sourceforge.net/pyphant/?rev=674&view=rev
Author: zklaus
Date: 2010-04-06 10:09:55 +0000 (Tue, 06 Apr 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Testcases for FMF version 1.1 introduced.
TestFMFLoader checks physical constants being adapted to new CODATA recommendations.
TestFMFLoader checks the correct interpretation of FMF 1.0 files.
Enh. documentation of quantities and quantities.PhysicalQuantities modules.
Included PhysicalQuantities module from revision 7dfc86beb3c71b25438bfb89b3e242bc9e3fca3c in order to distinct between FMF 1.0 and FMF 1.1 files.
Adapted units parsec and galUS to version 1.1 of FMF.
Abbreviated definition of Hartree and added missing invcm.
Renamed abbreviation of Avogadro number to NA.
Changed abbreviation of unified atomic mass unit to 'u'.
Added Rydberg constant.
Corrected constants Fa, e, me, mp, Nav, and k due to CODATA-2006 recommendataion.
Bugfix: module PhysicalQuantities is now named quantities
Introduced Faraday constant Fa
Added Faradays constant.
Renamed abbriviation of the gravitational constant to 'G' and corrected the value with respect to the CODATA 2006 recommendation.
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/DataContainer.py
trunk/src/pyphant/pyphant/core/FieldContainer.py
trunk/src/pyphant/pyphant/quantities/ParseQuantities.py
trunk/src/pyphant/pyphant/quantities/__init__.py
trunk/src/pyphant/pyphant/tests/TestParseQuantities.py
trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py
trunk/src/workers/OSC/OSC/CompareFields.py
trunk/src/workers/OSC/OSC/ComputeFunctional.py
trunk/src/workers/OSC/OSC/ErrorEstimator.py
trunk/src/workers/OSC/OSC/EstimateParameter.py
trunk/src/workers/OSC/OSC/ExtremumFinder.py
trunk/src/workers/OSC/OSC/OscCurrent.py
trunk/src/workers/OSC/OSC/Smoother.py
trunk/src/workers/OSC/OSC/ThicknessModeller.py
trunk/src/workers/fmfile/fmfile/FMFLoader.py
trunk/src/workers/fmfile/fmfile/tests/TestFMFLoader.py
Added Paths:
-----------
trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py
Modified: trunk/src/pyphant/pyphant/core/DataContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/DataContainer.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/pyphant/pyphant/core/DataContainer.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
-# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
+# Copyright (c) 2009-2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -196,7 +196,7 @@
\t .data \t- Table of samples stored in a numpy.ndarray.
\t .desc \t- Description numpy.dtype of the ndarray.
-\t .units \t- List of quantities.objects denoting the units of
+\t .units \t- List of quantities objects denoting the units of
\t\t\t the columns.
\t .longname \t- Notation of the data, e.g. 'database query',
\t\t\t which is used for the automatic annotation of charts.
Modified: trunk/src/pyphant/pyphant/core/FieldContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/FieldContainer.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/pyphant/pyphant/core/FieldContainer.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -2,7 +2,7 @@
from __future__ import with_statement
# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
-# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
+# Copyright (c) 2009-2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/pyphant/pyphant/quantities/ParseQuantities.py
===================================================================
--- trunk/src/pyphant/pyphant/quantities/ParseQuantities.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/pyphant/pyphant/quantities/ParseQuantities.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -40,8 +40,12 @@
import mx.DateTime.ISO
from pyphant.quantities import Quantity
+from pyphant.quantities.PhysicalQuantities import PhysicalQuantity
-def str2unit(unit):
+import logging
+_logger = logging.getLogger("pyphant")
+
+def str2unit(unit,FMFversion='1.1'):
"""The function str2unit returns either a quantity or a float from a given string."""
# Prepare conversion to quantity
if unit.startswith('.'):
@@ -59,14 +63,22 @@
elif not (unit[0].isdigit() or unit[0]=='-'):
unit = '1'+unit
# Convert input to quantity or float
- try:
+ if FMFversion not in ['1.0','1.1']:
+ raise ValueError, 'FMFversion %s not supported.' % FMFversion
+ else:
unit = unit.replace('^', '**')
- unit = Quantity(unit.encode('utf-8'))
- except:
- unit = float(unit)
+ try:
+ unit = unit.replace('^', '**')
+ if FMFversion=='1.1':
+ unit = Quantity(unit.encode('utf-8'))
+ elif FMFversion=='1.0':
+ unit1_0 = PhysicalQuantity(unit.encode('utf-8'))
+ unit = Quantity(str(unit1_0.inBaseUnits()))
+ except:
+ unit = float(unit)
return unit
-def parseQuantity(value):
+def parseQuantity(value,FMFversion='1.1'):
import re
pm = re.compile(ur"(?:\\pm|\+-|\+/-)")
try:
@@ -76,10 +88,10 @@
if value.startswith('('):
value = float(value[1:])
error, unit = [s.strip() for s in error.split(')')]
- unit = str2unit(unit)
+ unit = str2unit(unit,FMFversion)
value *= unit
else:
- value = str2unit(value)
+ value = str2unit(value,FMFversion)
if error != None:
if error.endswith('%'):
error = value*float(error[:-1])/100.0
@@ -87,17 +99,17 @@
try:
error = float(error)*unit
except:
- error = str2unit(error)
+ error = str2unit(error,FMFversion)
return value, error
-def parseVariable(oldVal):
+def parseVariable(oldVal,FMFversion='1.1'):
shortname, value = tuple([s.strip() for s in oldVal.split('=')])
- value, error = parseQuantity(value)
+ value, error = parseQuantity(value,FMFversion)
return (shortname, value, error)
-def parseDateTime(value):
+def parseDateTime(value,FMFversion='1.1'):
"""
- >>>parseDateTime('2004-08-21 12:00:00+-12h')
+ >>>parseDateTime('2004-08-21 12:00:00+-12hr')
(Quantity(731814.5,'d'), Quantity(0.5,'d'))
>>>parseDateTime('2004-08-21 12:00:00')
(Quantity(731814.5,'d'), None)
@@ -105,7 +117,11 @@
datetimeWithError = value.split('+-')
if len(datetimeWithError)==2:
datetime = mx.DateTime.ISO.ParseAny(datetimeWithError[0])
- error = parseQuantity(datetimeWithError[1])[0].inUnitsOf('d')
+ uncertainty = parseQuantity(datetimeWithError[1],FMFversion)[0]
+ if uncertainty.isCompatible('h'):
+ _logger.warning("The uncertainty of timestamp %s has the unit 'h', which is deprecated. The correct abbreviation for hour is 'hr'." % value)
+ uncertainty = uncertainty*Quantity('1hr/h')
+ error = uncertainty.inUnitsOf('d')
else:
datetime = mx.DateTime.ISO.ParseAny(value)
error = None
Copied: trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py (from rev 673, trunk/src/pyphant/pyphant/quantities/__init__.py)
===================================================================
--- trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py (rev 0)
+++ trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -0,0 +1,1004 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 1998-2007, Konrad Hinsen <hi...@cn...>
+# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2010, Andreas W. Liehr <li...@us...>
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# * Neither the name of the Freiburg Materials Research Center,
+# University of Freiburg nor the names of its contributors may be used to
+# endorse or promote products derived from this software without specific
+# prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
+# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+Quantities with units
+
+based on the module Scientific.Physics.PhysicalQuantities
+written by Conrad Hinsen with contributions from Greg Ward.
+A comprehensive documentation of these units is given in
+Riede et al: On the Communication of Scientific Results:
+The Full-Metadata Format (http://arxiv.org/abs/0904.1299),
+which defines version 1.0 of the Full-Metadata Format.
+
+This module provides a data type that represents a physical
+quantity together with its unit. It is possible to add and
+subtract these quantities if the units are compatible, and
+a quantity can be converted to another compatible unit.
+Multiplication, subtraction, and raising to integer powers
+is allowed without restriction, and the result will have
+the correct unit. A quantity can be raised to a non-integer
+power only if the result can be represented by integer powers
+of the base units.
+
+The values of physical constants are taken from the 1986
+recommended values from CODATA. Other conversion factors
+(e.g. for British units) come from various sources. I can't
+guarantee for the correctness of all entries in the unit
+table, so use this at your own risk.
+"""
+
+rc = { 'fetchCurrencyRates' : False }
+
+class NumberDict(dict):
+
+ """
+ Dictionary storing numerical values
+
+ Constructor: NumberDict()
+
+ An instance of this class acts like an array of number with
+ generalized (non-integer) indices. A value of zero is assumed
+ for undefined entries. NumberDict instances support addition,
+ and subtraction with other NumberDict instances, and multiplication
+ and division by scalars.
+ """
+
+ def __getitem__(self, item):
+ try:
+ return dict.__getitem__(self, item)
+ except KeyError:
+ return 0
+
+ def __coerce__(self, other):
+ if type(other) == type({}):
+ other = NumberDict(other)
+ return self, other
+
+ def __add__(self, other):
+ sum_dict = NumberDict()
+ for key in self.keys():
+ sum_dict[key] = self[key]
+ for key in other.keys():
+ sum_dict[key] = sum_dict[key] + other[key]
+ return sum_dict
+
+ def __sub__(self, other):
+ sum_dict = NumberDict()
+ for key in self.keys():
+ sum_dict[key] = self[key]
+ for key in other.keys():
+ sum_dict[key] = sum_dict[key] - other[key]
+ return sum_dict
+
+ def __mul__(self, other):
+ new = NumberDict()
+ for key in self.keys():
+ new[key] = other*self[key]
+ return new
+ __rmul__ = __mul__
+
+ def __div__(self, other):
+ new = NumberDict()
+ for key in self.keys():
+ new[key] = self[key]/other
+ return new
+
+import numpy.oldnumeric
+def int_sum(a, axis=0):
+ return numpy.oldnumeric.add.reduce(a, axis)
+def zeros_st(shape, other):
+ return numpy.oldnumeric.zeros(shape, dtype=other.dtype)
+from numpy import ndarray as array_type
+
+
+import re, string
+
+
+# Class definitions
+
+class PhysicalQuantity:
+
+ """
+ Physical quantity with units
+
+ PhysicalQuantity instances allow addition, subtraction,
+ multiplication, and division with each other as well as
+ multiplication, division, and exponentiation with numbers.
+ Addition and subtraction check that the units of the two operands
+ are compatible and return the result in the units of the first
+ operand. A limited set of mathematical functions (from module
+ Numeric) is applicable as well:
+
+ - sqrt: equivalent to exponentiation with 0.5.
+
+ - sin, cos, tan: applicable only to objects whose unit is
+ compatible with 'rad'.
+
+ See the documentation of the PhysicalQuantities module for a list
+ of the available units.
+
+ Here is an example on usage:
+
+ >>> from PhysicalQuantities import PhysicalQuantity as p # short hand
+ >>> distance1 = p('10 m')
+ >>> distance2 = p('10 km')
+ >>> total = distance1 + distance2
+ >>> total
+ PhysicalQuantity(10010.0,'m')
+ >>> total.convertToUnit('km')
+ >>> total.getValue()
+ 10.01
+ >>> total.getUnitName()
+ 'km'
+ >>> total = total.inBaseUnits()
+ >>> total
+ PhysicalQuantity(10010.0,'m')
+ >>>
+ >>> t = p(314159., 's')
+ >>> # convert to days, hours, minutes, and second:
+ >>> t2 = t.inUnitsOf('d','h','min','s')
+ >>> t2_print = ' '.join([str(i) for i in t2])
+ >>> t2_print
+ '3.0 d 15.0 h 15.0 min 59.0 s'
+ >>>
+ >>> e = p('2.7 Hartree*Nav')
+ >>> e.convertToUnit('kcal/mol')
+ >>> e
+ PhysicalQuantity(1694.2757596034764,'kcal/mol')
+ >>> e = e.inBaseUnits()
+ >>> str(e)
+ '7088849.77818 kg*m**2/s**2/mol'
+ >>>
+ >>> freeze = p('0 degC')
+ >>> freeze = freeze.inUnitsOf ('degF')
+ >>> str(freeze)
+ '32.0 degF'
+ >>>
+ """
+
+ def __init__(self, *args):
+ """
+ There are two constructor calling patterns:
+
+ 1. PhysicalQuantity(value, unit), where value is any number
+ and unit is a string defining the unit
+
+ 2. PhysicalQuantity(value_with_unit), where value_with_unit
+ is a string that contains both the value and the unit,
+ i.e. '1.5 m/s'. This form is provided for more convenient
+ interactive use.
+
+ @param args: either (value, unit) or (value_with_unit,)
+ @type args: (number, C{str}) or (C{str},)
+ """
+ if len(args) == 2:
+ self.value = args[0]
+ self.unit = _findUnit(args[1])
+ else:
+ s = string.strip(args[0])
+ match = PhysicalQuantity._number.match(s)
+ if match is None:
+ raise TypeError('No number found')
+ self.value = string.atof(match.group(0))
+ self.unit = _findUnit(s[len(match.group(0)):])
+
+ _number = re.compile('[+-]?[0-9]+(\\.[0-9]*)?([eE][+-]?[0-9]+)?')
+
+ def __str__(self):
+ return str(self.value) + ' ' + self.unit.name()
+
+ def __repr__(self):
+ return (self.__class__.__name__ + '(' + `self.value` + ',' +
+ `self.unit.name()` + ')')
+
+ def _sum(self, other, sign1, sign2):
+ if not isPhysicalQuantity(other):
+ raise TypeError('Incompatible types')
+ new_value = sign1*self.value + \
+ sign2*other.value*other.unit.conversionFactorTo(self.unit)
+ return self.__class__(new_value, self.unit)
+
+ def __add__(self, other):
+ return self._sum(other, 1, 1)
+
+ __radd__ = __add__
+
+ def __sub__(self, other):
+ return self._sum(other, 1, -1)
+
+ def __rsub__(self, other):
+ return self._sum(other, -1, 1)
+
+ def __cmp__(self, other):
+ diff = self._sum(other, 1, -1)
+ return cmp(diff.value, 0)
+
+ def __mul__(self, other):
+ if not isPhysicalQuantity(other):
+ return self.__class__(self.value*other, self.unit)
+ value = self.value*other.value
+ unit = self.unit*other.unit
+ if unit.isDimensionless():
+ return value*unit.factor
+ else:
+ return self.__class__(value, unit)
+
+ __rmul__ = __mul__
+
+ def __div__(self, other):
+ if not isPhysicalQuantity(other):
+ return self.__class__(self.value/other, self.unit)
+ value = self.value/other.value
+ unit = self.unit/other.unit
+ if unit.isDimensionless():
+ return value*unit.factor
+ else:
+ return self.__class__(value, unit)
+
+ def __rdiv__(self, other):
+ if not isPhysicalQuantity(other):
+ return self.__class__(other/self.value, pow(self.unit, -1))
+ value = other.value/self.value
+ unit = other.unit/self.unit
+ if unit.isDimensionless():
+ return value*unit.factor
+ else:
+ return self.__class__(value, unit)
+
+ def __pow__(self, other):
+ if isPhysicalQuantity(other):
+ raise TypeError('Exponents must be dimensionless')
+ return self.__class__(pow(self.value, other), pow(self.unit, other))
+
+ def __rpow__(self, other):
+ raise TypeError('Exponents must be dimensionless')
+
+ def __abs__(self):
+ return self.__class__(abs(self.value), self.unit)
+
+ def __pos__(self):
+ return self
+
+ def __neg__(self):
+ return self.__class__(-self.value, self.unit)
+
+ def __nonzero__(self):
+ return self.value != 0
+
+ def convertToUnit(self, unit):
+ """
+ Change the unit and adjust the value such that
+ the combination is equivalent to the original one. The new unit
+ must be compatible with the previous unit of the object.
+
+ @param unit: a unit
+ @type unit: C{str}
+ @raise TypeError: if the unit string is not a know unit or a
+ unit incompatible with the current one
+ """
+ unit = _findUnit(unit)
+ self.value = _convertValue (self.value, self.unit, unit)
+ self.unit = unit
+
+ def inUnitsOf(self, *units):
+ """
+ Express the quantity in different units. If one unit is
+ specified, a new PhysicalQuantity object is returned that
+ expresses the quantity in that unit. If several units
+ are specified, the return value is a tuple of
+ PhysicalObject instances with with one element per unit such
+ that the sum of all quantities in the tuple equals the the
+ original quantity and all the values except for the last one
+ are integers. This is used to convert to irregular unit
+ systems like hour/minute/second.
+
+ @param units: one or several units
+ @type units: C{str} or sequence of C{str}
+ @returns: one or more physical quantities
+ @rtype: L{PhysicalQuantity} or C{tuple} of L{PhysicalQuantity}
+ @raises TypeError: if any of the specified units are not compatible
+ with the original unit
+ """
+ units = map(_findUnit, units)
+ if len(units) == 1:
+ unit = units[0]
+ value = _convertValue (self.value, self.unit, unit)
+ return self.__class__(value, unit)
+ else:
+ units.sort()
+ result = []
+ value = self.value
+ unit = self.unit
+ for i in range(len(units)-1,-1,-1):
+ value = value*unit.conversionFactorTo(units[i])
+ if i == 0:
+ rounded = value
+ else:
+ rounded = _round(value)
+ result.append(self.__class__(rounded, units[i]))
+ value = value - rounded
+ unit = units[i]
+ return tuple(result)
+
+ # Contributed by Berthold Hoellmann
+ def inBaseUnits(self):
+ """
+ @returns: the same quantity converted to base units,
+ i.e. SI units in most cases
+ @rtype: L{PhysicalQuantity}
+ """
+ new_value = self.value * self.unit.factor
+ num = ''
+ denom = ''
+ for i in xrange(9):
+ unit = _base_names[i]
+ power = self.unit.powers[i]
+ if power < 0:
+ denom = denom + '/' + unit
+ if power < -1:
+ denom = denom + '**' + str(-power)
+ elif power > 0:
+ num = num + '*' + unit
+ if power > 1:
+ num = num + '**' + str(power)
+ if len(num) == 0:
+ num = '1'
+ else:
+ num = num[1:]
+ return self.__class__(new_value, num + denom)
+
+ def isCompatible (self, unit):
+ """
+ @param unit: a unit
+ @type unit: C{str}
+ @returns: C{True} if the specified unit is compatible with the
+ one of the quantity
+ @rtype: C{bool}
+ """
+ unit = _findUnit (unit)
+ return self.unit.isCompatible (unit)
+
+ def getValue(self):
+ """Return value (float) of physical quantity (no unit)."""
+ return self.value
+
+ def getUnitName(self):
+ """Return unit (string) of physical quantity."""
+ return self.unit.name()
+
+ def sqrt(self):
+ return pow(self, 0.5)
+
+ def sin(self):
+ if self.unit.isAngle():
+ return numpy.oldnumeric.sin(self.value * \
+ self.unit.conversionFactorTo(_unit_table['rad']))
+ else:
+ raise TypeError('Argument of sin must be an angle')
+
+ def cos(self):
+ if self.unit.isAngle():
+ return numpy.oldnumeric.cos(self.value * \
+ self.unit.conversionFactorTo(_unit_table['rad']))
+ else:
+ raise TypeError('Argument of cos must be an angle')
+
+ def tan(self):
+ if self.unit.isAngle():
+ return numpy.oldnumeric.tan(self.value * \
+ self.unit.conversionFactorTo(_unit_table['rad']))
+ else:
+ raise TypeError('Argument of tan must be an angle')
+
+
+class PhysicalUnit:
+
+ """
+ Physical unit
+
+ A physical unit is defined by a name (possibly composite), a scaling
+ factor, and the exponentials of each of the SI base units that enter into
+ it. Units can be multiplied, divided, and raised to integer powers.
+ """
+
+ def __init__(self, names, factor, powers, offset=0):
+ """
+ @param names: a dictionary mapping each name component to its
+ associated integer power (e.g. C{{'m': 1, 's': -1}})
+ for M{m/s}). As a shorthand, a string may be passed
+ which is assigned an implicit power 1.
+ @type names: C{dict} or C{str}
+ @param factor: a scaling factor
+ @type factor: C{float}
+ @param powers: the integer powers for each of the nine base units
+ @type powers: C{list} of C{int}
+ @param offset: an additive offset to the base unit (used only for
+ temperatures)
+ @type offset: C{float}
+ """
+ if type(names) == type(''):
+ self.names = NumberDict()
+ self.names[names] = 1
+ else:
+ self.names = names
+ self.factor = factor
+ self.offset = offset
+ self.powers = powers
+
+ def __repr__(self):
+ return '<PhysicalUnit ' + self.name() + '>'
+
+ __str__ = __repr__
+
+ def __cmp__(self, other):
+ if self.powers != other.powers:
+ raise TypeError('Incompatible units')
+ return cmp(self.factor, other.factor)
+
+ def __mul__(self, other):
+ if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
+ raise TypeError("cannot multiply units with non-zero offset")
+ if isPhysicalUnit(other):
+ return PhysicalUnit(self.names+other.names,
+ self.factor*other.factor,
+ map(lambda a,b: a+b,
+ self.powers, other.powers))
+ else:
+ return PhysicalUnit(self.names+{str(other): 1},
+ self.factor*other,
+ self.powers,
+ self.offset * other)
+
+ __rmul__ = __mul__
+
+ def __div__(self, other):
+ if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
+ raise TypeError("cannot divide units with non-zero offset")
+ if isPhysicalUnit(other):
+ return PhysicalUnit(self.names-other.names,
+ self.factor/other.factor,
+ map(lambda a,b: a-b,
+ self.powers, other.powers))
+ else:
+ return PhysicalUnit(self.names+{str(other): -1},
+ self.factor/other, self.powers)
+
+ def __rdiv__(self, other):
+ if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
+ raise TypeError("cannot divide units with non-zero offset")
+ if isPhysicalUnit(other):
+ return PhysicalUnit(other.names-self.names,
+ other.factor/self.factor,
+ map(lambda a,b: a-b,
+ other.powers, self.powers))
+ else:
+ return PhysicalUnit({str(other): 1}-self.names,
+ other/self.factor,
+ map(lambda x: -x, self.powers))
+
+ def __pow__(self, other):
+ if self.offset != 0:
+ raise TypeError("cannot exponentiate units with non-zero offset")
+ if isinstance(other, int):
+ return PhysicalUnit(other*self.names, pow(self.factor, other),
+ map(lambda x,p=other: x*p, self.powers))
+ if isinstance(other, float):
+ inv_exp = 1./other
+ rounded = int(numpy.oldnumeric.floor(inv_exp+0.5))
+ if abs(inv_exp-rounded) < 1.e-10:
+ if reduce(lambda a, b: a and b,
+ map(lambda x, e=rounded: x%e == 0, self.powers)):
+ f = pow(self.factor, other)
+ p = map(lambda x,p=rounded: x/p, self.powers)
+ if reduce(lambda a, b: a and b,
+ map(lambda x, e=rounded: x%e == 0,
+ self.names.values())):
+ names = self.names/rounded
+ else:
+ names = NumberDict()
+ if f != 1.:
+ names[str(f)] = 1
+ for i in range(len(p)):
+ names[_base_names[i]] = p[i]
+ return PhysicalUnit(names, f, p)
+ else:
+ raise TypeError('Illegal exponent')
+ raise TypeError('Only integer and inverse integer exponents allowed')
+
+ def conversionFactorTo(self, other):
+ """
+ @param other: another unit
+ @type other: L{PhysicalUnit}
+ @returns: the conversion factor from this unit to another unit
+ @rtype: C{float}
+ @raises TypeError: if the units are not compatible
+ """
+ if self.powers != other.powers:
+ raise TypeError('Incompatible units')
+ if self.offset != other.offset and self.factor != other.factor:
+ raise TypeError(('Unit conversion (%s to %s) cannot be expressed ' +
+ 'as a simple multiplicative factor') % \
+ (self.name(), other.name()))
+ return self.factor/other.factor
+
+ def conversionTupleTo(self, other): # added 1998/09/29 GPW
+ """
+ @param other: another unit
+ @type other: L{PhysicalUnit}
+ @returns: the conversion factor and offset from this unit to
+ another unit
+ @rtype: (C{float}, C{float})
+ @raises TypeError: if the units are not compatible
+ """
+ if self.powers != other.powers:
+ raise TypeError('Incompatible units')
+
+ # let (s1,d1) be the conversion tuple from 'self' to base units
+ # (ie. (x+d1)*s1 converts a value x from 'self' to base units,
+ # and (x/s1)-d1 converts x from base to 'self' units)
+ # and (s2,d2) be the conversion tuple from 'other' to base units
+ # then we want to compute the conversion tuple (S,D) from
+ # 'self' to 'other' such that (x+D)*S converts x from 'self'
+ # units to 'other' units
+ # the formula to convert x from 'self' to 'other' units via the
+ # base units is (by definition of the conversion tuples):
+ # ( ((x+d1)*s1) / s2 ) - d2
+ # = ( (x+d1) * s1/s2) - d2
+ # = ( (x+d1) * s1/s2 ) - (d2*s2/s1) * s1/s2
+ # = ( (x+d1) - (d1*s2/s1) ) * s1/s2
+ # = (x + d1 - d2*s2/s1) * s1/s2
+ # thus, D = d1 - d2*s2/s1 and S = s1/s2
+ factor = self.factor / other.factor
+ offset = self.offset - (other.offset * other.factor / self.factor)
+ return (factor, offset)
+
+ def isCompatible (self, other): # added 1998/10/01 GPW
+ """
+ @param other: another unit
+ @type other: L{PhysicalUnit}
+ @returns: C{True} if the units are compatible, i.e. if the powers of
+ the base units are the same
+ @rtype: C{bool}
+ """
+ return self.powers == other.powers
+
+ def isDimensionless(self):
+ return not reduce(lambda a,b: a or b, self.powers)
+
+ def isAngle(self):
+ return self.powers[7] == 1 and \
+ reduce(lambda a,b: a + b, self.powers) == 1
+
+ def setName(self, name):
+ self.names = NumberDict()
+ self.names[name] = 1
+
+ def name(self):
+ num = ''
+ denom = ''
+ for unit in self.names.keys():
+ power = self.names[unit]
+ if power < 0:
+ denom = denom + '/' + unit
+ if power < -1:
+ denom = denom + '**' + str(-power)
+ elif power > 0:
+ num = num + '*' + unit
+ if power > 1:
+ num = num + '**' + str(power)
+ if len(num) == 0:
+ num = '1'
+ else:
+ num = num[1:]
+ return num + denom
+
+
+# Type checks
+
+def isPhysicalUnit(x):
+ """
+ @param x: an object
+ @type x: any
+ @returns: C{True} if x is a L{PhysicalUnit}
+ @rtype: C{bool}
+ """
+ return hasattr(x, 'factor') and hasattr(x, 'powers')
+
+def isPhysicalQuantity(x):
+ """
+ @param x: an object
+ @type x: any
+ @returns: C{True} if x is a L{PhysicalQuantity}
+ @rtype: C{bool}
+ """
+ return hasattr(x, 'value') and hasattr(x, 'unit')
+
+
+# Helper functions
+
+def _findUnit(unit):
+ if type(unit) == type(''):
+ name = string.strip(unit)
+ unit = eval(name, _unit_table)
+ for cruft in ['__builtins__', '__args__']:
+ try: del _unit_table[cruft]
+ except: pass
+
+ if not isPhysicalUnit(unit):
+ raise TypeError(str(unit) + ' is not a unit')
+ return unit
+
+def _round(x):
+ if numpy.oldnumeric.greater(x, 0.):
+ return numpy.oldnumeric.floor(x)
+ else:
+ return numpy.oldnumeric.ceil(x)
+
+
+def _convertValue (value, src_unit, target_unit):
+ (factor, offset) = src_unit.conversionTupleTo(target_unit)
+ return (value + offset) * factor
+
+
+# SI unit definitions
+
+_base_names = ['m', 'kg', 's', 'A', 'K', 'mol', 'cd', 'rad', 'sr','EUR']
+
+_base_units = [('m', PhysicalUnit('m', 1., [1,0,0,0,0,0,0,0,0,0])),
+ ('g', PhysicalUnit('g', 0.001, [0,1,0,0,0,0,0,0,0,0])),
+ ('s', PhysicalUnit('s', 1., [0,0,1,0,0,0,0,0,0,0])),
+ ('A', PhysicalUnit('A', 1., [0,0,0,1,0,0,0,0,0,0])),
+ ('K', PhysicalUnit('K', 1., [0,0,0,0,1,0,0,0,0,0])),
+ ('mol', PhysicalUnit('mol', 1., [0,0,0,0,0,1,0,0,0,0])),
+ ('cd', PhysicalUnit('cd', 1., [0,0,0,0,0,0,1,0,0,0])),
+ ('rad', PhysicalUnit('rad', 1., [0,0,0,0,0,0,0,1,0,0])),
+ ('sr', PhysicalUnit('sr', 1., [0,0,0,0,0,0,0,0,1,0])),
+ ('EUR', PhysicalUnit('EUR', 1., [0,0,0,0,0,0,0,0,0,1])),
+ ]
+
+_prefixes = [('Y', 1.e24),
+ ('Z', 1.e21),
+ ('E', 1.e18),
+ ('P', 1.e15),
+ ('T', 1.e12),
+ ('G', 1.e9),
+ ('M', 1.e6),
+ ('k', 1.e3),
+ ('h', 1.e2),
+ ('da', 1.e1),
+ ('d', 1.e-1),
+ ('c', 1.e-2),
+ ('m', 1.e-3),
+ ('mu', 1.e-6),
+ ('n', 1.e-9),
+ ('p', 1.e-12),
+ ('f', 1.e-15),
+ ('a', 1.e-18),
+ ('z', 1.e-21),
+ ('y', 1.e-24),
+ ]
+
+_unit_table = {}
+
+for unit in _base_units:
+ _unit_table[unit[0]] = unit[1]
+
+_help = []
+
+def _addUnit(name, unit, comment=''):
+ if _unit_table.has_key(name):
+ raise KeyError, 'Unit ' + name + ' already defined'
+ if comment:
+ _help.append((name, comment, unit))
+ if type(unit) == type(''):
+ unit = eval(unit, _unit_table)
+ for cruft in ['__builtins__', '__args__']:
+ try: del _unit_table[cruft]
+ except: pass
+ unit.setName(name)
+ _unit_table[name] = unit
+
+def _addPrefixed(unit):
+ _help.append('Prefixed units for %s:' % unit)
+ _prefixed_names = []
+ if unit in ['EUR']:
+ validPrefixes = filter(lambda prefix: prefix[1]>=1000,_prefixes)
+ else:
+ validPrefixes = _prefixes
+ for prefix in validPrefixes:
+ name = prefix[0] + unit
+ _addUnit(name, prefix[1]*_unit_table[unit])
+ _prefixed_names.append(name)
+ _help.append(', '.join(_prefixed_names))
+
+# SI derived units; these automatically get prefixes
+_help.append('SI derived units; these automatically get prefixes:\n' + \
+ ', '.join([prefix + ' (%.0E)' % value for prefix, value in _prefixes]) + \
+ '\n')
+
+
+_unit_table['kg'] = PhysicalUnit('kg', 1., [0,1,0,0,0,0,0,0,0,0])
+
+_addUnit('Hz', '1/s', 'Hertz')
+_addUnit('N', 'm*kg/s**2', 'Newton')
+_addUnit('Pa', 'N/m**2', 'Pascal')
+_addUnit('J', 'N*m', 'Joule')
+_addUnit('W', 'J/s', 'Watt')
+_addUnit('C', 's*A', 'Coulomb')
+_addUnit('V', 'W/A', 'Volt')
+_addUnit('F', 'C/V', 'Farad')
+_addUnit('ohm', 'V/A', 'Ohm')
+_addUnit('S', 'A/V', 'Siemens')
+_addUnit('Wb', 'V*s', 'Weber')
+_addUnit('T', 'Wb/m**2', 'Tesla')
+_addUnit('H', 'Wb/A', 'Henry')
+_addUnit('lm', 'cd*sr', 'Lumen')
+_addUnit('lx', 'lm/m**2', 'Lux')
+_addUnit('Bq', '1/s', 'Becquerel')
+_addUnit('Gy', 'J/kg', 'Gray')
+_addUnit('Sv', 'J/kg', 'Sievert')
+
+del _unit_table['kg']
+
+for unit in _unit_table.keys():
+ _addPrefixed(unit)
+
+# Fundamental constants
+_help.append('Fundamental constants:')
+
+_unit_table['pi'] = numpy.oldnumeric.pi
+_addUnit('c', '299792458.*m/s', 'speed of light')
+_addUnit('mu0', '4.e-7*pi*N/A**2', 'permeability of vacuum')
+_addUnit('eps0', '1/mu0/c**2', 'permittivity of vacuum')
+_addUnit('Grav', '6.67259e-11*m**3/kg/s**2', 'gravitational constant')
+_addUnit('hplanck', '6.6260755e-34*J*s', 'Planck constant')
+_addUnit('hbar', 'hplanck/(2*pi)', 'Planck constant / 2pi')
+_addUnit('e', '1.60217733e-19*C', 'elementary charge')
+_addUnit('me', '9.1093897e-31*kg', 'electron mass')
+_addUnit('mp', '1.6726231e-27*kg', 'proton mass')
+_addUnit('Nav', '6.0221367e23/mol', 'Avogadro number')
+_addUnit('k', '1.380658e-23*J/K', 'Boltzmann constant')
+
+# Time units
+_help.append('Time units:')
+
+_addUnit('min', '60*s', 'minute')
+_addUnit('h', '60*min', 'hour')
+_addUnit('d', '24*h', 'day')
+_addUnit('wk', '7*d', 'week')
+_addUnit('yr', '365.25*d', 'year')
+
+# Length units
+_help.append('Length units:')
+
+_addUnit('inch', '2.54*cm', 'inch')
+_addUnit('ft', '12*inch', 'foot')
+_addUnit('yd', '3*ft', 'yard')
+_addUnit('mi', '5280.*ft', '(British) mile')
+_addUnit('nmi', '1852.*m', 'Nautical mile')
+_addUnit('Ang', '1.e-10*m', 'Angstrom')
+_addUnit('lyr', 'c*yr', 'light year')
+_addUnit('AU', '149597870691*m', 'astronomical unit')
+_addUnit('pc', '3.08567758128E16*m','parsec')
+_addUnit('Bohr', '4*pi*eps0*hbar**2/me/e**2', 'Bohr radius')
+
+# Area units
+_help.append('Area units:')
+
+_addUnit('ha', '10000*m**2', 'hectare')
+_addUnit('acres', 'mi**2/640', 'acre')
+_addUnit('b', '1.e-28*m', 'barn')
+
+# Volume units
+_help.append('Volume units:')
+
+_addUnit('l', 'dm**3', 'liter')
+_addUnit('dl', '0.1*l', 'deci liter')
+_addUnit('cl', '0.01*l', 'centi liter')
+_addUnit('ml', '0.001*l', 'milli liter')
+_addUnit('tsp', '4.92892159375*ml', 'teaspoon')
+_addUnit('tbsp', '3*tsp', 'tablespoon')
+_addUnit('floz', '2*tbsp', 'fluid ounce')
+_addUnit('cup', '8*floz', 'cup')
+_addUnit('pt', '16*floz', 'pint')
+_addUnit('qt', '2*pt', 'quart')
+_addUnit('galUS', '4*qt', 'US gallon')
+_addUnit('galUK', '4.54609*l', 'British gallon')
+
+# Mass units
+_help.append('Mass units:')
+
+_addUnit('amu', '1.6605402e-27*kg', 'atomic mass units')
+_addUnit('oz', '28.349523125*g', 'ounce')
+_addUnit('lb', '16*oz', 'pound')
+_addUnit('ton', '2000*lb', 'ton')
+
+# Concentration units
+_help.append('Concentration units:')
+_addUnit('M','mol/m**3','molar concentration')
+_addUnit('mM','0.001*mol/m**3','millimolar')
+_addUnit('muM','10**-6*mol/m**3','micromolar')
+
+# Force units
+_help.append('Force units:')
+
+_addUnit('dyn', '1.e-5*N', 'dyne (cgs unit)')
+
+# Energy units
+_help.append('Energy units:')
+
+_addUnit('erg', '1.e-7*J', 'erg (cgs unit)')
+_addUnit('eV', 'e*V', 'electron volt')
+_addUnit('Hartree', 'me*e**4/16/pi**2/eps0**2/hbar**2', 'Wavenumbers/inverse cm')
+_addUnit('Ken', 'k*K', 'Kelvin as energy unit')
+_addUnit('cal', '4.184*J', 'thermochemical calorie')
+_addUnit('kcal', '1000*cal', 'thermochemical kilocalorie')
+_addUnit('cali', '4.1868*J', 'international calorie')
+_addUnit('kcali', '1000*cali', 'international kilocalorie')
+_addUnit('Btu', '1055.05585262*J', 'British thermal unit')
+
+_addPrefixed('eV')
+
+# Power units
+_help.append('Power units:')
+
+_addUnit('hp', '745.7*W', 'horsepower')
+
+# Pressure units
+_help.append('Pressure units:')
+
+_addUnit('bar', '1.e5*Pa', 'bar (cgs unit)')
+_addUnit('dbar', '1.e4*Pa', 'dbar (cgs unit)')
+_addUnit('mbar', '1.e2*Pa', 'mbar (cgs unit)')
+_addUnit('atm', '101325.*Pa', 'standard atmosphere')
+_addUnit('torr', 'atm/760', 'torr = mm of mercury')
+_addUnit('psi', '6894.75729317*Pa', 'pounds per square inch')
+
+# Angle units
+_help.append('Angle units:')
+
+_addUnit('deg', 'pi*rad/180', 'degrees')
+
+_help.append('Temperature units:')
+# Temperature units -- can't use the 'eval' trick that _addUnit provides
+# for degC and degF because you can't add units
+kelvin = _findUnit ('K')
+_addUnit ('degR', '(5./9.)*K', 'degrees Rankine')
+_addUnit ('degC', PhysicalUnit (None, 1.0, kelvin.powers, 273.15),
+ 'degrees Celcius')
+_addUnit ('degF', PhysicalUnit (None, 5./9., kelvin.powers, 459.67),
+ 'degree Fahrenheit')
+del kelvin
+
+_help.append('Old European currencies:')
+#Taken from http://www.xe.com/euro.php on 2008-11-05
+_addUnit('ATS', 'EUR/13.7603' ,'Austria, Schilling')
+_addUnit('BEF', 'EUR/40.3399' ,'Belgium, Franc')
+_addUnit('CYP', 'EUR/0.585274','Cyprus, Pound')
+_addUnit('DEM', 'EUR/1.95583' ,'Germany, Deutsche Mark')
+_addUnit('ESP', 'EUR/166.386' ,'Spain, Peseta')
+_addUnit('FIM', 'EUR/5.94573' ,'Finland, Markka')
+_addUnit('FRF', 'EUR/5.94573' ,'France, Franc')
+_addUnit('GRD', 'EUR/340.750' ,'Greece, Drachma')
+_addUnit('IEP', 'EUR/0.787564','Ireland, Pound')
+_addUnit('ITL', 'EUR/1936.27' ,'Italy, Lira')
+_addUnit('LUF', 'EUR/40.3399' ,'Luxembourg, Franc')
+_addUnit('MTL', 'EUR/0.429300','Malta, Lira')
+_addUnit('NLG', 'EUR/2.20371' ,'The Netherlands, Guilder (also called Florin)')
+_addUnit('PTE', 'EUR/200.482' ,'Portugal, Escudo')
+_addUnit('SIT', 'EUR/239.640' ,'Slovenia, Tolar')
+_addUnit('VAL', 'EUR/1936.27' ,'Vatican City, Lira')
+
+#Get daily updated exchange rates
+if rc['fetchCurrencyRates']:
+ import urllib
+ from xml.dom import minidom
+ url = "http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml"
+ currencyNames={'USD':'US dollar' , 'JPY':'Japanese yen',
+ 'BGN':'Bulgarian lev' , 'CZK':'Czech koruna',
+ 'DKK':'Danish krone' , 'EEK':'Estonian kroon',
+ 'GBP':'Pound sterling' ,'HUF':'Hungarian forint',
+ 'LTL':'Lithuanian litas','LVL':'Latvian lats',
+ 'PLN':'Polish zloty', 'RON':'New Romanian leu',
+ 'SEK':'Swedish krona', 'SKK':'Slovak koruna',
+ 'CHF':'Swiss franc', 'ISK':'Icelandic krona',
+ 'NOK':'Norwegian krone', 'HRK':'Croatian kuna',
+ 'RUB':'Russian rouble', 'TRY':'New Turkish lira',
+ 'AUD':'Australian dollar','BRL':'Brasilian real',
+ 'CAD':'Canadian dollar', 'CNY':'Chinese yuan renminbi',
+ 'HKD':'Hong Kong dollar','IDR':'Indonesian rupiah',
+ 'KRW':'South Korean won','MXN':'Mexican peso',
+ 'MYR':'Malaysian ringgit','NZD':'New Zealand dollar',
+ 'PHP':'Philippine peso', 'SGD':'Singapore dollar',
+ 'THB':'Thai baht', 'ZAR':'South African rand'}
+ try:
+ doc = minidom.parseString(urllib.urlopen(url).read())
+ elements = doc.documentElement.getElementsByTagName('Cube')
+ for element in elements[2:]:
+ currency = element.getAttribute('currency').encode('utf8')
+ _addUnit(currency,
+ 'EUR/%s' % element.getAttribute('rate').encode('utf8'),
+ currencyNames[currency])
+ print "Added exchange rate of %s for %s." % (elements[1].getAttribute('time'),
+ [ i.getAttribute('currency').encode('utf8')
+ for i in elements[2:] ])
+ except:
+ print "WARNING: No daily exchange rates available."
+
+def description():
+ """Return a string describing all available units."""
+ s = '' # collector for description text
+ for entry in _help:
+ if isinstance(entry, basestring):
+ # headline for new section
+ s += '\n' + entry + '\n'
+ elif isinstance(entry, tuple):
+ name, comment, unit = entry
+ s += '%-8s %-26s %s\n' % (name, comment, unit)
+ else:
+ # impossible
+ raise TypeError, 'wrong construction of _help list'
+ return s
+
+# add the description of the units to the module's doc string:
+__doc__ += '\n' + description()
+
+# Some demonstration code. Run with "python -i PhysicalQuantities.py"
+# to have this available.
+
+if __name__ == '__main__':
+
+# from Scientific.N import *
+ l = PhysicalQuantity(10., 'm')
+ big_l = PhysicalQuantity(10., 'km')
+ print big_l + l
+ t = PhysicalQuantity(314159., 's')
+ print t.inUnitsOf('d','h','min','s')
+
+ p = PhysicalQuantity # just a shorthand...
+
+ e = p('2.7 Hartree*Nav')
+ e.convertToUnit('kcal/mol')
+ print e
+ print e.inBaseUnits()
+
+ freeze = p('0 degC')
+ print freeze.inUnitsOf ('degF')
+
+ euro = PhysicalQuantity('1 EUR')
+ print euro.inUnitsOf('DEM')
+ print euro.inUnitsOf('USD')
+
+ euroSQM = PhysicalQuantity('19.99 EUR/m**2')
+ print "%s=%s" % (euroSQM,euroSQM.inUnitsOf('EUR/cm**2'))
Modified: trunk/src/pyphant/pyphant/quantities/__init__.py
===================================================================
--- trunk/src/pyphant/pyphant/quantities/__init__.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/pyphant/pyphant/quantities/__init__.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -2,7 +2,7 @@
# Copyright (c) 1998-2007, Konrad Hinsen <hi...@cn...>
# Copyright (c) 2008-2010, Rectorate of the University of Freiburg
-# Copyright (c) 2009-2010, Andreas W. Liehr
+# Copyright (c) 2009-2010, Andreas W. Liehr <li...@us...>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -36,6 +36,11 @@
based on the module Scientific.Physics.PhysicalQuantities
written by Conrad Hinsen with contributions from Greg Ward.
+A comprehensive documentation of these units is given in
+Riede et al: On the Communication of Scientific Data:
+The Full-Metadata Format, Computer Physics and Communication 181(3),
+2010, pp 651--662 (doi:10.1016/j.cpc.2009.11.014),
+which defines version 1.1 of the Full-Metadata Format.
This module provides a data type that represents a physical
quantity together with its unit. It is possible to add and
@@ -126,7 +131,7 @@
class Quantity:
"""
- Physical quantity with units
+ Quantity with units
Quantity instances allow addition, subtraction,
multiplication, and division with each other as well as
@@ -141,12 +146,12 @@
- sin, cos, tan: applicable only to objects whose unit is
compatible with 'rad'.
- See the documentation of the quantities.module for a list
+ See the documentation of the quantities module for a list
of the available units.
Here is an example on usage:
- >>> from quantities.import Quantity as p # short hand
+ >>> from quantities import Quantity as p # short hand
>>> distance1 = p('10 m')
>>> distance2 = p('10 km')
>>> total = distance1 + distance2
@@ -163,12 +168,12 @@
>>>
>>> t = p(314159., 's')
>>> # convert to days, hours, minutes, and second:
- >>> t2 = t.inUnitsOf('d','h','min','s')
+ >>> t2 = t.inUnitsOf('d','hr','min','s')
>>> t2_print = ' '.join([str(i) for i in t2])
>>> t2_print
'3.0 d 15.0 h 15.0 min 59.0 s'
>>>
- >>> e = p('2.7 Hartree*Nav')
+ >>> e = p('2.7 Hartree*NA')
>>> e.convertToUnit('kcal/mol')
>>> e
Quantity(1694.2757596034764,'kcal/mol')
@@ -237,7 +242,8 @@
return self._sum(other, -1, 1)
def __cmp__(self, other):
- diff = self._sum(other, 1, -1)
+ normed = self.inBaseUnits() # Minimizing numerical errors
+ diff = normed._sum(other, 1, -1)
return cmp(diff.value, 0)
def __mul__(self, other):
@@ -791,21 +797,23 @@
_addUnit('c', '299792458.*m/s', 'speed of light')
_addUnit('mu0', '4.e-7*pi*N/A**2', 'permeability of vacuum')
_addUnit('eps0', '1/mu0/c**2', 'permittivity of vacuum')
-_addUnit('Grav', '6.67259e-11*m**3/kg/s**2', 'gravitational constant')
-_addUnit('hplanck', '6.6260755e-34*J*s', 'Planck constant')
-_addUnit('hbar', 'hplanck/(2*pi)', 'Planck constant / 2pi')
-_addUnit('e', '1.60217733e-19*C', 'elementary charge')
-_addUnit('me', '9.1093897e-31*kg', 'electron mass')
-_addUnit('mp', '1.6726231e-27*kg', 'proton mass')
-_addUnit('Nav', '6.0221367e23/mol', 'Avogadro number')
-_addUnit('k', '1.380658e-23*J/K', 'Boltzmann constant')
+_addUnit('Fa','96485.3399*C/mol', 'Faraday constant')
+_addUnit('G', '6.67428e-11*m**3/kg/s**2', 'gravitational constant')
+_addUnit('h', '6.62606896e-34*J*s', 'Planck constant')
+_addUnit('hbar', 'h/(2*pi)', 'Planck constant / 2pi')
+_addUnit('e', '1.602176487e-19*C', 'elementary charge')
+_addUnit('me', '9.10938215e-31*kg', 'electron mass')
+_addUnit('mp', '1.672621637e-27*kg', 'proton mass')
+_addUnit('NA', '6.02214179e23/mol', 'Avogadro number')
+_addUnit('k', '1.3806504e-23*J/K', 'Boltzmann constant')
+_addUnit('Ryd','10973731.568527/m','Rydberg constant')
# Time units
_help.append('Time units:')
_addUnit('min', '60*s', 'minute')
-_addUnit('h', '60*min', 'hour')
-_addUnit('d', '24*h', 'day')
+_addUnit('hr', '60*min', 'hour')
+_addUnit('d', '24*hr', 'day')
_addUnit('wk', '7*d', 'week')
_addUnit('yr', '365.25*d', 'year')
@@ -820,7 +828,7 @@
_addUnit('Ang', '1.e-10*m', 'Angstrom')
_addUnit('lyr', 'c*yr', 'light year')
_addUnit('AU', '149597870691*m', 'astronomical unit')
-_addUnit('pc', '3.08567758128E16*m','parsec')
+_addUnit('pc', '3.0856776E16*m','parsec')
_addUnit('Bohr', '4*pi*eps0*hbar**2/me/e**2', 'Bohr radius')
# Area units
@@ -843,13 +851,13 @@
_addUnit('cup', '8*floz', 'cup')
_addUnit('pt', '16*floz', 'pint')
_addUnit('qt', '2*pt', 'quart')
-_addUnit('galUS', '4*qt', 'US gallon')
+_addUnit('galUS', '231*inch**3', 'US gallon')
_addUnit('galUK', '4.54609*l', 'British gallon')
# Mass units
_help.append('Mass units:')
-_addUnit('amu', '1.6605402e-27*kg', 'atomic mass units')
+_addUnit('u', '1.660538782e-27*kg', 'atomic mass units')
_addUnit('oz', '28.349523125*g', 'ounce')
_addUnit('lb', '16*oz', 'pound')
_addUnit('ton', '2000*lb', 'ton')
@@ -870,7 +878,8 @@
_addUnit('erg', '1.e-7*J', 'erg (cgs unit)')
_addUnit('eV', 'e*V', 'electron volt')
-_addUnit('Hartree', 'me*e**4/16/pi**2/eps0**2/hbar**2', 'Wavenumbers/inverse cm')
+_addUnit('Hartree', 'me*e**4/eps0**2/h**2/4', 'Hartree')
+_addUnit('invcm', 'h*c/cm', 'Wave-numbers/inverse cm')
_addUnit('Ken', 'k*K', 'Kelvin as energy unit')
_addUnit('cal', '4.184*J', 'thermochemical calorie')
_addUnit('kcal', '1000*cal', 'thermochemical kilocalorie')
@@ -984,21 +993,18 @@
# add the description of the units to the module's doc string:
__doc__ += '\n' + description()
-# Some demonstration code. Run with "python -i quantities.py"
-# to have this available.
+# Some demonstration code.
if __name__ == '__main__':
-
-# from Scientific.N import *
l = Quantity(10., 'm')
big_l = Quantity(10., 'km')
print big_l + l
t = Quantity(314159., 's')
- print t.inUnitsOf('d','h','min','s')
+ print t.inUnitsOf('d','hr','min','s')
p = Quantity # just a shorthand...
- e = p('2.7 Hartree*Nav')
+ e = p('2.7 Hartree*NA')
e.convertToUnit('kcal/mol')
print e
print e.inBaseUnits()
Modified: trunk/src/pyphant/pyphant/tests/TestParseQuantities.py
===================================================================
--- trunk/src/pyphant/pyphant/tests/TestParseQuantities.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/pyphant/pyphant/tests/TestParseQuantities.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -40,13 +40,18 @@
from pyphant.quantities.ParseQuantities import parseDateTime,str2unit
from pyphant.quantities import Quantity
"""
- >>>parseDateTime('2004-08-21 12:00:00+-12h')
+ >>>parseDateTime('2004-08-21 12:00:00+-12hr')
(Quantity(731814.5,'d'), Quantity(0.5,'d'))
>>>parseDateTime('2004-08-21 12:00:00')
(Quantity(731814.5,'d'), None)
"""
class TestParseDateTime(unittest.TestCase):
- def testWithUncertainty(self):
+ def testWithError(self):
+ self.assertEqual(parseDateTime('2004-08-21 12:00:00+-12hr'),
+ (Quantity(731814.5,'d'), Quantity(0.5,'d'))
+ )
+
+ def testWithErrorOldDeprecatedAbbreviation(self):
self.assertEqual(parseDateTime('2004-08-21 12:00:00+-12h'),
(Quantity(731814.5,'d'), Quantity(0.5,'d'))
)
@@ -59,6 +64,21 @@
result = str2unit('1V')
self.assertEqual(expected,result)
+ def setUp(self):
+ self.inputDict = {'complexJ':'1.0j','Joule':'1.0J'}
+
+ def testJouleValue(self):
+ """Physical quantities with unit Joule are indicated by 'J'."""
+ result = str2unit(self.inputDict['Joule'])
+ self.assertEqual(result,Quantity(self.inputDict['Joule']))
+
+ def testHourPlanck(self):
+ """In FMF 1.0 unit 'h' denotes hours, while in FMF 1.1 'h' denotes the Planck constant."""
+ result = str2unit('1h')
+ self.assertEqual(result,Quantity('6.62606896e-34 J*s'))
+ result = str2unit('1h',FMFversion='1.0')
+ self.assertEqual(result,Quantity('3600s'))
+
if __name__ == "__main__":
import sys
if len(sys.argv) == 1:
Modified: trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py
===================================================================
--- trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -152,10 +152,10 @@
vol = (self.yxCube & fslice).getVolume()
if not isQuantity(vol):
vol = float(vol)
- yxratio = 2.0 * vol / (fslice.getVolume() + self.yxCube.getVolume())
+ yxratio = vol / fslice.getVolume()
fszCube = Cube([slice(zvalue - self.ztol, zvalue + self.ztol)])
zvol = (self.zCube & fszCube).getVolume()
- # weird notation necessary for PhysicalQuantities
+ # weird notation necessary for Quantities
if yxratio >= self.boundRatio and zvol != 0 * zvol:
orCube = self.yxCube | fslice
self.yxCube = orCube
Modified: trunk/src/workers/OSC/OSC/CompareFields.py
===================================================================
--- trunk/src/workers/OSC/OSC/CompareFields.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/CompareFields.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/workers/OSC/OSC/ComputeFunctional.py
===================================================================
--- trunk/src/workers/OSC/OSC/ComputeFunctional.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/ComputeFunctional.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -46,7 +47,6 @@
from pyphant import quantities
import copy
-
class ComputeFunctional(Worker.Worker):
API = 2
VERSION = 1
Modified: trunk/src/workers/OSC/OSC/ErrorEstimator.py
===================================================================
--- trunk/src/workers/OSC/OSC/ErrorEstimator.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/ErrorEstimator.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2007-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/workers/OSC/OSC/EstimateParameter.py
===================================================================
--- trunk/src/workers/OSC/OSC/EstimateParameter.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/EstimateParameter.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -44,7 +45,6 @@
from pyphant import quantities
import copy
-
class EstimateParameter(Worker.Worker):
API = 2
VERSION = 1
Modified: trunk/src/workers/OSC/OSC/ExtremumFinder.py
===================================================================
--- trunk/src/workers/OSC/OSC/ExtremumFinder.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/ExtremumFinder.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -2,6 +2,7 @@
#!Pyphant's ExtremumFinder worker
#!-------------------------------
# Copyright (c) 2007-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/workers/OSC/OSC/OscCurrent.py
===================================================================
--- trunk/src/workers/OSC/OSC/OscCurrent.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/OscCurrent.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,4 +1,3 @@
-
# -*- coding: utf-8 -*-
# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
Modified: trunk/src/workers/OSC/OSC/Smoother.py
===================================================================
--- trunk/src/workers/OSC/OSC/Smoother.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/Smoother.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2007-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/workers/OSC/OSC/ThicknessModeller.py
===================================================================
--- trunk/src/workers/OSC/OSC/ThicknessModeller.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/OSC/OSC/ThicknessModeller.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
Modified: trunk/src/workers/fmfile/fmfile/FMFLoader.py
===================================================================
--- trunk/src/workers/fmfile/fmfile/FMFLoader.py 2010-03-20 21:59:38 UTC (rev 673)
+++ trunk/src/workers/fmfile/fmfile/FMFLoader.py 2010-04-06 10:09:55 UTC (rev 674)
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
-# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
+# Copyright (c) 2009-2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -325,7 +325,7 @@
def readSingleFile(b, pixelName):
_logger.info(u"Parsing file %s." % pixelName)
- preParsedData, d = preParseData(b)
+ preParsedData, d, FMFversion = preParseData(b)
from configobj import ConfigObj,ConfigObjError
class FMFConfigObj(ConfigObj):
_keyword = re.compile(r'''^ # line start
@@ -344,7 +344,7 @@
except ConfigObjError,e:
from sys import exit
exit('%s\nPlease check the syntax of the FMF-file, in particular the correct usage of comments.' % e)
- return config2tables(preParsedData, config)
+ return config2tables(preParsedData, config,FMFversion)
def parseBool(value):
if value.lower() == 'true':
@@ -353,32 +353,34 @@
return False
raise AttributeError
-_converters = [
- int,
- float,
- parseBool,
- parseVariable,
- parseQuantity,
- complex, # Complex is checked after variables and quantities,
+def getConverters(FMFversion='1.1'):
+ converters = [
+ int,
+ float,
+ parseBool,
+ lambda v: parseVariable(v,FMFversion),
+ lambda q: parseQuantity(q,FMFversion),
+ complex, # Complex is checked after variables and quantities,
# because 1J is 1 Joule and not an imaginary number.
- parse...
[truncated message content] |
|
From: <zk...@us...> - 2010-03-20 21:59:44
|
Revision: 673
http://pyphant.svn.sourceforge.net/pyphant/?rev=673&view=rev
Author: zklaus
Date: 2010-03-20 21:59:38 +0000 (Sat, 20 Mar 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Fixed stupid typo.
Modified Paths:
--------------
trunk/doc/demo/viewOSC.py
Modified: trunk/doc/demo/viewOSC.py
===================================================================
--- trunk/doc/demo/viewOSC.py 2010-03-18 11:41:19 UTC (rev 672)
+++ trunk/doc/demo/viewOSC.py 2010-03-20 21:59:38 UTC (rev 673)
@@ -199,7 +199,7 @@
label ="$\\Delta%s$"%minimaPos.shortname, linestyle='dashed')
pylab.vlines(maximaPos.data[:,index],0.1,1.0,
label ="$%s$"%maximaPos.shortname)
- pylab.vlines(minimaPos.data[:,index]+maximaPos.error[:,index],0.1,1.0,
+ pylab.vlines(maximaPos.data[:,index]+maximaPos.error[:,index],0.1,1.0,
label ="$\\Delta%s$"%maximaPos.shortname, linestyle='dashed')
pylab.vlines(maximaPos.data[:,index]-maximaPos.error[:,index],0.1,1.0,
label ="$\\Delta%s$"%maximaPos.shortname, linestyle='dashed')
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-03-18 11:41:27
|
Revision: 672
http://pyphant.svn.sourceforge.net/pyphant/?rev=672&view=rev
Author: zklaus
Date: 2010-03-18 11:41:19 +0000 (Thu, 18 Mar 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Enh: Added maxima to mra anaylsis and incorporated it into the parameter estimation.
Modified Paths:
--------------
trunk/doc/demo/viewOSC.py
trunk/src/workers/OSC/OSC/ComputeFunctional.py
trunk/src/workers/OSC/OSC/EstimateParameter.py
trunk/src/workers/OSC/OSC/MRA.py
trunk/src/workers/OSC/OSC/tests/TestMRA.py
Modified: trunk/doc/demo/viewOSC.py
===================================================================
--- trunk/doc/demo/viewOSC.py 2010-03-14 14:26:07 UTC (rev 671)
+++ trunk/doc/demo/viewOSC.py 2010-03-18 11:41:19 UTC (rev 672)
@@ -171,7 +171,8 @@
worker = recipe.getWorker("ThicknessModeller")
simulation = worker.plugCalcAbsorption.getResult()
worker = recipe.getWorker("MRA Exp")
- minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
+ minimaPos = worker.plugMra.getResult()[r'\lambda_{min}'].inUnitsOf(simulation.dimensions[1])
+ maximaPos = worker.plugMra.getResult()[r'\lambda_{max}'].inUnitsOf(simulation.dimensions[1])
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
xPos = table[u"x-position"]
@@ -196,6 +197,12 @@
label ="$\\Delta%s$"%minimaPos.shortname, linestyle='dashed')
pylab.vlines(minimaPos.data[:,index]-minimaPos.error[:,index],0.1,1.0,
label ="$\\Delta%s$"%minimaPos.shortname, linestyle='dashed')
+ pylab.vlines(maximaPos.data[:,index],0.1,1.0,
+ label ="$%s$"%maximaPos.shortname)
+ pylab.vlines(minimaPos.data[:,index]+maximaPos.error[:,index],0.1,1.0,
+ label ="$\\Delta%s$"%maximaPos.shortname, linestyle='dashed')
+ pylab.vlines(maximaPos.data[:,index]-maximaPos.error[:,index],0.1,1.0,
+ label ="$\\Delta%s$"%maximaPos.shortname, linestyle='dashed')
pylab.title(result)
pylab.xlabel(simulation.dimensions[1].label)
@@ -205,7 +212,8 @@
worker = recipe.getWorker("ThicknessModeller")[0]
simulation = worker.plugCalcAbsorption.getResult()
worker = recipe.getWorker("MRA Exp")
- minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
+ minimaPos = worker.plugMra.getResult()[r'\lambda_{min}'].inUnitsOf(simulation.dimensions[1])
+ maximaPos = worker.plugMra.getResult()[r'\lambda_{max}'].inUnitsOf(simulation.dimensions[1])
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
xPos = table[u"x-position"]
@@ -222,6 +230,8 @@
if not noIndicators:
pylab.vlines(minimaPos.data[:,index],0.1,1.0,
label ="$%s$"%minimaPos.shortname)
+ pylab.vlines(maximaPos.data[:,index],0.1,1.0,
+ label ="$%s$"%maximaPos.shortname)
pylab.title(result)
pylab.xlabel(simulation.dimensions[1].label)
pylab.ylabel(simulation.label)
@@ -239,11 +249,12 @@
def functional(recipe, curvNo, noIndicators):
worker = recipe.getWorker("Compute Functional")
- functional = worker.plugCompute.getResult()
+ functional = worker.plugCompute.getResult()[r'F_{\lambda_{min}}']
worker = recipe.getWorker("ThicknessModeller")
simulation = worker.plugCalcAbsorption.getResult()
worker = recipe.getWorker("MRA Exp")
- minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
+ minimaPos = worker.plugMra.getResult()[r'\lambda_{min}'].inUnitsOf(simulation.dimensions[1])
+ maximaPos = worker.plugMra.getResult()[r'\lambda_{max}'].inUnitsOf(simulation.dimensions[1])
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
thickness = table[u"thickness"]
@@ -253,6 +264,8 @@
ordinate = functional.dimensions[1].data
pylab.hlines(minimaPos.data[:,index],ordinate.min(),ordinate.max(),
label ="$%s$"%minimaPos.shortname)
+ pylab.hlines(maximaPos.data[:,index],ordinate.min(),ordinate.max(),
+ label ="$%s$"%maximaPos.shortname)
abscissae = functional.dimensions[0].data
pylab.vlines(thickness.data[index],abscissae.min(),abscissae.max(),
label ="$%s$"%minimaPos.shortname)
@@ -265,7 +278,8 @@
thickness = table[u"thickness"]
index = curvNo2Index(table[u"pixel"], curvNo)
worker = recipe.getWorker("MRA Exp")
- minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
+ minimaPos = worker.plugMra.getResult()[r'\lambda_{min}'].inUnitsOf(simulation.dimensions[1])
+ maximaPos = worker.plugMra.getResult()[r'\lambda_{max}'].inUnitsOf(simulation.dimensions[1])
visualizer = ImageVisualizer(simulation, False)
ordinate = simulation.dimensions[1].data
if not noIndicators:
@@ -274,6 +288,8 @@
abscissae = simulation.dimensions[0].data
pylab.vlines(minimaPos.data[:,index],abscissae.min(),abscissae.max(),
label ="$%s$"%minimaPos.shortname)
+ pylab.vlines(maximaPos.data[:,index],abscissae.min(),abscissae.max(),
+ label ="$%s$"%maximaPos.shortname)
def dumpMinima(recipe):
worker = recipe.getWorker("ThicknessModeller")
@@ -286,7 +302,7 @@
thickness = table[u"thickness"]
cols = [index, xPos, yPos, thickness]
worker = recipe.getWorker("MRA Exp")
- minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
+ minimaPos = worker.plugMra.getResult()[r'\lambda_{min}'].inUnitsOf(simulation.dimensions[1])
import numpy
data = numpy.vstack([ c.data for c in cols]
+ numpy.vsplit(minimaPos.data, len(minimaPos.data))).transpose()
Modified: trunk/src/workers/OSC/OSC/ComputeFunctional.py
===================================================================
--- trunk/src/workers/OSC/OSC/ComputeFunctional.py 2010-03-14 14:26:07 UTC (rev 671)
+++ trunk/src/workers/OSC/OSC/ComputeFunctional.py 2010-03-18 11:41:19 UTC (rev 672)
@@ -53,7 +53,7 @@
REVISION = "$Revision$"[11:-1]
name = "Compute Functional"
- _sockets = [("field", Connectors.TYPE_IMAGE)]
+ _sockets = [("field", Connectors.TYPE_ARRAY)]
_params = [("extentX", u"Extension of x-axis [%%]", 10, None),
("extentY", u"Extension of y-axis [%%]", 10, None)]
@@ -92,13 +92,15 @@
subscriber %= percentage
result = DataContainer.FieldContainer(functional.transpose(),
dimensions=[yDim, xDim],
- longname = 'functional',
- shortname= 'F'
+ longname = 'functional of %s'%field.longname,
+ shortname= 'F_{%s}'%field.shortname
)
return result
- @Worker.plug(Connectors.TYPE_IMAGE)
+ @Worker.plug(Connectors.TYPE_ARRAY)
def compute(self, field, subscriber=1):
- functional = self.computeDistances(field, subscriber)
- functional.seal()
- return functional
+ functionals = DataContainer.SampleContainer([self.computeDistances(column, subscriber) for column in field],
+ longname='Functionals of %s'%field.longname,
+ shortname='F_{%s}'%field.shortname)
+ functionals.seal()
+ return functionals
Modified: trunk/src/workers/OSC/OSC/EstimateParameter.py
===================================================================
--- trunk/src/workers/OSC/OSC/EstimateParameter.py 2010-03-14 14:26:07 UTC (rev 671)
+++ trunk/src/workers/OSC/OSC/EstimateParameter.py 2010-03-18 11:41:19 UTC (rev 672)
@@ -51,20 +51,34 @@
REVISION = "$Revision$"[11:-1]
name = "Estimate Parameter"
- _sockets = [("model", Connectors.TYPE_IMAGE),
- ("experimental", Connectors.TYPE_IMAGE)]
- _params = [("extentX", u"Extension of x-axis [%%]", 10, None),
+ _sockets = [("model", Connectors.TYPE_ARRAY),
+ ("experimental", Connectors.TYPE_ARRAY)]
+ _params = [("minima_model", u"Minima in the model", [u"Minima"], None),
+ ("maxima_model", u"Maxima in the model", [u"Maxima"], None),
+ ("minima_experimental", u"Minima in the experiment", [u"Minima"], None),
+ ("maxima_experimental", u"Maxima in the experiment", [u"Maxima"], None),
+ ("extentX", u"Extension of x-axis [%%]", 10, None),
("extentY", u"Extension of y-axis [%%]", 10, None)]
- def calculateThickness(self, row, model, error=None):
+ def refreshParams(self, subscriber=None):
+ if self.socketModel.isFull():
+ templ = self.socketModel.getResult( subscriber )
+ self.paramMinima_model.possibleValues = templ.longnames.keys()
+ self.paramMaxima_model.possibleValues = templ.longnames.keys()
+ if self.socketExperimental.isFull():
+ templ = self.socketExperimental.getResult( subscriber )
+ self.paramMinima_experimental.possibleValues = templ.longnames.keys()
+ self.paramMaxima_experimental.possibleValues = templ.longnames.keys()
+
+ def calculateThickness(self, row_minima, row_maxima, minima_model, maxima_model,
+ minima_error=None, maxima_error=None):
"""
Given a vector of minima (row) and a 2 dimensional model,
this estimates the corresponding parameter.
"""
- if len(row)==0:
+ if len(row_minima)+len(row_maxima)==0:
return numpy.nan
- data = model.data.transpose()
- def calc(row, col, error):
+ def calc(row, model, col, error):
if error:
weight=0
for c,e in zip(row, error):
@@ -78,38 +92,68 @@
else:
return sum([col[numpy.argmin((model.dimensions[0].data-c)**2)]
for c in row])
- i = numpy.argmin(numpy.array([calc(row, col, error) for col in data]))
- return model.dimensions[1].data[i]
+ minima_data = minima_model.data.transpose()
+ maxima_data = maxima_model.data.transpose()
+ i = numpy.argmin(numpy.array([calc(row_minima, minima_model, minima_col, minima_error)
+ +calc(row_maxima, maxima_model, maxima_col, maxima_error)
+ for minima_col, maxima_col in
+ zip(minima_data, maxima_data)]))
+ assert(minima_model.dimensions[1].data[i]==maxima_model.dimensions[1].data[i])
+ return minima_model.dimensions[1].data[i]
@Worker.plug(Connectors.TYPE_IMAGE)
def compute(self, model, experimental, subscriber=1):
- renormedExp = experimental.inUnitsOf(model.dimensions[0])
- minima = renormedExp.data.transpose()
- if renormedExp.error != None:
- error = iter(renormedExp.error.transpose())
+ minima_model = model[self.paramMinima_model.value]
+ maxima_model = model[self.paramMaxima_model.value]
+ minima_experimental = experimental[
+ self.paramMinima_experimental.value
+ ].inUnitsOf(
+ minima_model.dimensions[0]
+ )
+ maxima_experimental = experimental[
+ self.paramMaxima_experimental.value
+ ].inUnitsOf(
+ maxima_model.dimensions[0]
+ )
+ minima = minima_experimental.data.transpose()
+ if minima_experimental.error != None:
+ minima_error = iter(minima_experimental.error.transpose())
else:
- error = None
+ minima_error = None
+ maxima = maxima_experimental.data.transpose()
+ if maxima_experimental.error != None:
+ maxima_error = iter(maxima_experimental.error.transpose())
+ else:
+ maxima_error = None
parameter = []
inc = 100.0/float(len(minima))
acc = inc
subscriber %= acc
- for row in minima:
- if error:
- filteredError = filter(
- lambda c: not numpy.isnan(c), error.next())
+ for row_minima, row_maxima in zip(minima, maxima):
+ if minima_error:
+ filtered_minima_error = filter(
+ lambda c: not numpy.isnan(c), minima_error.next())
else:
- filteredError = None
+ filtered_minima_error = None
+ if maxima_error:
+ filtered_maxima_error = filter(
+ lambda c: not numpy.isnan(c), maxima_error.next())
+ else:
+ filtered_maxima_error = None
parameter.append(self.calculateThickness(
- filter(lambda c: not numpy.isnan(c), row),
- model,
- filteredError
+ filter(lambda c: not numpy.isnan(c), row_minima),
+ filter(lambda c: not numpy.isnan(c), row_maxima),
+ minima_model,
+ maxima_model,
+ filtered_minima_error,
+ filtered_maxima_error
))
acc += inc
subscriber %= acc
result = DataContainer.FieldContainer(
numpy.array(parameter),
- longname = model.dimensions[-1].longname,
- shortname = model.dimensions[-1].shortname,
- unit = model.dimensions[-1].unit)
+ longname = minima_model.dimensions[-1].longname,
+ shortname = minima_model.dimensions[-1].shortname,
+ unit = minima_model.dimensions[-1].unit)
result.seal()
return result
Modified: trunk/src/workers/OSC/OSC/MRA.py
===================================================================
--- trunk/src/workers/OSC/OSC/MRA.py 2010-03-14 14:26:07 UTC (rev 671)
+++ trunk/src/workers/OSC/OSC/MRA.py 2010-03-18 11:41:19 UTC (rev 672)
@@ -1,8 +1,9 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2008-2010, Rectorate of the University of Freiburg
# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
-# All rights reserved.
+# Copyright (c) 2010, Klaus Zimmermann (zk...@us...)
+# all rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
@@ -49,12 +50,29 @@
from scipy.special import ive
from scipy.signal import convolve
+def findMaxima(fieldData, lastExtrema=None):
+ leftLess = fieldData[:-2] < fieldData[1:-1]
+ leftEqual = fieldData[:-2] == fieldData[1:-1]
+ rightLess = fieldData[2:] < fieldData[1:-1]
+ rightEqual = fieldData[2:] == fieldData[1:-1]
+ maxima_c = numpy.logical_and(leftLess, rightLess)
+ maxima_le = numpy.logical_and(leftLess, rightEqual)
+ maxima_re = numpy.logical_and(rightLess, leftEqual)
+ maxima_e = numpy.logical_and(maxima_le[:-1], maxima_re[1:])
+ maxima = numpy.logical_or(maxima_c[1:], maxima_e).nonzero()[0]
+ if lastExtrema==None or len(maxima)==0 or len(lastExtrema)==len(maxima):
+ return maxima
+ trackedMaxima = []
+ for lastMaximum in lastExtrema:
+ distance = (maxima-lastMaximum)**2
+ trackedMaxima.append(distance.argmin())
+ return maxima[trackedMaxima]
+
def findMinima(fieldData, lastExtrema=None):
leftGreater = fieldData[:-2] > fieldData[1:-1]
leftEqual = fieldData[:-2] == fieldData[1:-1]
rightGreater = fieldData[2:] > fieldData[1:-1]
rightEqual = fieldData[2:] == fieldData[1:-1]
-
minima_c = numpy.logical_and(leftGreater, rightGreater)
minima_le = numpy.logical_and(leftGreater, rightEqual)
minima_re = numpy.logical_and(rightGreater, leftEqual)
@@ -75,17 +93,6 @@
kernel = ive(numpy.arange(-n, n), sigma)
return convolve(field.data, kernel, mode='same')
-#def mra1d(dim, field, n):
-# mrr = [convolveMRA(field, sigma) for sigma in
-# numpy.linspace(1, n,10).tolist()]
-# mrr.insert(0,field.data)
-# firstMinima = lastMinima = findMinima(mrr[-1], None)
-# for row in reversed(mrr[:-1]):
-# lastMinima = findMinima(row, lastMinima)
-# pos = dim.data[numpy.array(lastMinima)+1]
-# error = numpy.abs(pos - dim.data[numpy.array(firstMinima)+1])
-# return pos, error
-
class MraError(RuntimeError):
def __init__(self, msg, convolvedField):
RuntimeError.__init__(self, msg)
@@ -95,20 +102,30 @@
sigmaSpace = numpy.linspace(n, 1, 10)
convolvedField = convolveMRA(field, sigmaSpace[0])
firstMinima = lastMinima = findMinima(convolvedField, None)
- if len(firstMinima)==0:
- raise MraError("No Minima found at sigma level %s"%sigmaSpace[0],
+ firstMaxima = lastMaxima = findMaxima(convolvedField, None)
+ if len(firstMinima)==0 and len(firstMaxima)==0:
+ raise MraError("No Extrema found at sigma level %s"%sigmaSpace[0],
convolvedField)
for sigma in sigmaSpace[1:]:
convolvedField = convolveMRA(field, sigma)
lastMinima = findMinima(convolvedField, lastMinima)
- if len(lastMinima)==0:
- import pylab
- pylab.plot(convolvedField)
- pylab.show()
- pos = dim.data[numpy.array(lastMinima)+1]
- error = numpy.abs(pos - dim.data[numpy.array(firstMinima)+1])
- return pos, error
+ lastMaxima = findMaxima(convolvedField, lastMaxima)
+ pos_minima = dim.data[numpy.array(lastMinima)+1]
+ error_minima = numpy.abs(pos_minima - dim.data[numpy.array(firstMinima)+1])
+ pos_maxima = dim.data[numpy.array(lastMaxima)+1]
+ error_maxima = numpy.abs(pos_maxima - dim.data[numpy.array(firstMaxima)+1])
+ return ((pos_minima, error_minima), (pos_maxima, error_maxima))
+def pos_error_to_data_container(p_e):
+ n = max(map(lambda (p,e): len(p), p_e))
+ m = len(p_e)
+ pos = numpy.ones((m,n),'float')*numpy.NaN
+ error = pos.copy()
+ for i in xrange(m):
+ for j in xrange(len(p_e[i][0])):
+ pos[i,j] = p_e[i][0][j]
+ error[i,j] = p_e[i][1][j]
+ return n, pos, error
class MRA(Worker.Worker):
API = 2
@@ -121,7 +138,7 @@
("longname",u"Name of result",'default',None),
("symbol",u"Symbol of result",'default',None)]
- @Worker.plug(Connectors.TYPE_IMAGE)
+ @Worker.plug(Connectors.TYPE_ARRAY)
def mra(self, field, subscriber=0):
dim = field.dimensions[-1]
try:
@@ -139,30 +156,36 @@
try:
p_e.append(mra1d(dim, field1d, sigmaMax))
except MraError:
- p_e.append(([],[]))
+ p_e.append((([],[]),([],[])))
acc += inc
subscriber %= acc
- n = max(map(lambda (p,e): len(p), p_e))
- m = len(p_e)
- pos = numpy.ones((m,n),'float')*numpy.NaN
- error = pos.copy()
- for i in xrange(m):
- for j in xrange(len(p_e[i][0])):
- pos[i,j] = p_e[i][0][j]
- error[i,j] = p_e[i][1][j]
- dims = [DataContainer.generateIndex(0,n), field.dimensions[0]]
+ minima, maxima = zip(*p_e)
+ n_min, pos_min, err_min = pos_error_to_data_container(minima)
+ n_max, pos_max, err_max = pos_error_to_data_container(maxima)
+ dims_min = [DataContainer.generateIndex(0,n_min), field.dimensions[0]]
+ dims_max = [DataContainer.generateIndex(0,n_max), field.dimensions[0]]
else:
- pos, error = mra1d(dim, field, sigmaMax)
- n = len(pos)
- dims = [DataContainer.generateIndex(0,n)]
+ (pos_min, err_min), (pos_max, err_max) = mra1d(dim, field, sigmaMax)
+ dims_min = [DataContainer.generateIndex(0,len(pos_min))]
+ dims_max = [DataContainer.generateIndex(0,len(pos_max))]
subscriber %= 100.
- roots = DataContainer.FieldContainer(pos.transpose(),
- error = error.transpose(),
- unit = dim.unit,
- dimensions = dims,
- mask = numpy.isnan(pos).transpose(),
- longname="%s of the local %s of %s" % (dim.longname,"minima",field.longname),
- shortname="%s_0" % dim.shortname)
+ minima = DataContainer.FieldContainer(pos_min.transpose(),
+ error = err_min.transpose(),
+ unit = dim.unit,
+ dimensions = dims_min,
+ mask = numpy.isnan(pos_min).transpose(),
+ longname="%s of the local %s of %s" % (dim.longname,"minima",field.longname),
+ shortname="%s_{min}" % dim.shortname)
+ maxima = DataContainer.FieldContainer(pos_max.transpose(),
+ error = err_max.transpose(),
+ unit = dim.unit,
+ dimensions = dims_max,
+ mask = numpy.isnan(pos_max).transpose(),
+ longname="%s of the local %s of %s" % (dim.longname,"maxima",field.longname),
+ shortname="%s_{max}" % dim.shortname)
+ roots = DataContainer.SampleContainer([minima, maxima],
+ longname="%s of the local %s of %s" % (dim.longname,"extrema",field.longname),
+ shortname="%s_{extrem}" % dim.shortname)
if self.paramLongname.value != 'default':
roots.longname = self.paramLongname.value
if self.paramSymbol.value != 'default':
Modified: trunk/src/workers/OSC/OSC/tests/TestMRA.py
===================================================================
--- trunk/src/workers/OSC/OSC/tests/TestMRA.py 2010-03-14 14:26:07 UTC (rev 671)
+++ trunk/src/workers/OSC/OSC/tests/TestMRA.py 2010-03-18 11:41:19 UTC (rev 672)
@@ -87,8 +87,26 @@
w.paramScale.value = "1.0m"
result = w.mra(self.V)
#Testing
- numpy.testing.assert_array_almost_equal(result.data,expectedResult.data,4)
+ numpy.testing.assert_array_almost_equal(result[r'x_{min}'].data,expectedResult.data,4)
+ def testMaxima(self):
+ """Test the correct computation of all local minima for a bistable potential."""
+ #Predict result
+ x0,curv,mask = fixedPoints(numpy.array([self.LAMBDA]),kappa1=self.kappa1)
+ expectedResult = DC.FieldContainer(numpy.extract(curv[0]>0,x0[0]),
+ unit = self.xField.unit,
+ longname = 'position of the local minima of electric potential',
+ shortname = 'x_0')
+ #Retrieve result from worker
+ w = MRA.MRA(None)
+ w.paramScale.value = "1.0m"
+ V = copy.deepcopy(self.V)
+ V.data*=-1
+ result = w.mra(V)
+ #Testing
+ numpy.testing.assert_array_almost_equal(result[r'x_{max}'].data,expectedResult.data,4)
+
+
class TestExtremumFinderTable(unittest.TestCase):
"""Sets up a mirror symmetric bistable potential with a continuous
distretisation and computes its local extrema and the respective
@@ -142,7 +160,7 @@
w = MRA.MRA(None)
w.paramScale.value = "1.0m"
#Retrieve result from worker
- result = copy.deepcopy(w.mra(self.V))
+ result = copy.deepcopy(w.mra(self.V))['x_{min}']
result.error=None
self.test(result,expectedResult,1e-2,1e-2)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-03-14 14:26:13
|
Revision: 671
http://pyphant.svn.sourceforge.net/pyphant/?rev=671&view=rev
Author: zklaus
Date: 2010-03-14 14:26:07 +0000 (Sun, 14 Mar 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Cosm: Nicer code formatting
Fix: Removed deficient ImageSaver visualizer
Fix: Removed inadequate batch workers
Modified Paths:
--------------
trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py
trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
trunk/src/workers/tools/tools/__init__.py
Modified: trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py
===================================================================
--- trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py 2010-02-25 17:31:27 UTC (rev 670)
+++ trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py 2010-03-14 14:26:07 UTC (rev 671)
@@ -152,17 +152,5 @@
pylab.ion()
pylab.show()
-class ImageSaver(object):
- name = 'Save Greyscale Image'
- def __init__(self, fieldContainer, show=True):
- self.fieldContainer = fieldContainer
- self.show = show
- #testing only:
- print("Enter filename: ")
- filename = raw_input()
- if filename != "":
- scipy.misc.imsave('/Users/aheld/CiSE/series/output/' + filename,
- fieldContainer.data)
DataVisReg.getInstance().registerVisualizer(TYPE_IMAGE, ImageVisualizer)
-DataVisReg.getInstance().registerVisualizer(TYPE_IMAGE, ImageSaver)
Modified: trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
===================================================================
--- trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-02-25 17:31:27 UTC (rev 670)
+++ trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-03-14 14:26:07 UTC (rev 671)
@@ -46,7 +46,8 @@
logging.basicConfig(level=logging.DEBUG,
filename=os.path.join(LOGDIR, u'pyphant.log'),
filemode='w',
- format="%(asctime)s - %(levelname)s:%(name)s:%(thread)d:%(module)s.%(funcName)s(l %(lineno)d):%(message)s")
+ format="%(asctime)s - %(levelname)s:%(name)s:%(thread)"\
+ "d:%(module)s.%(funcName)s(l %(lineno)d):%(message)s")
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
logging.getLogger('').addHandler(console)
@@ -63,6 +64,7 @@
import webbrowser
pltform = platform.system()
+
class wxPyphantApplication(wx.PySimpleApp):
def __init__(self, pathToRecipe=None):
self.pathToRecipe = pathToRecipe
@@ -71,12 +73,12 @@
def OnInit(self):
if not wx.PySimpleApp.OnInit(self):
return False
- self._logger=logging.getLogger("pyphant")
+ self._logger = logging.getLogger("pyphant")
self._excframe = wx.Frame(None, -2, "")
sys.excepthook = self.excepthook
sogl.SOGLInitialize()
self._knowledgeNode = None
- self._paramVisReg=ParamVisReg.ParamVisReg()
+ self._paramVisReg = ParamVisReg.ParamVisReg()
self._frame = wxPyphantFrame(self)
self._frame.Show()
return True
@@ -100,7 +102,9 @@
return self._frame
def configureWorker(self, worker):
- configureFrame=ConfigureFrame.ConfigureFrame(self._frame, self._paramVisReg, worker)
+ configureFrame = ConfigureFrame.ConfigureFrame(self._frame,
+ self._paramVisReg,
+ worker)
if configureFrame.ShowModal() == wx.ID_OK:
configureFrame.applyAll()
@@ -110,40 +114,38 @@
class wxPyphantFrame(wx.Frame):
- ID_WINDOW_TOP=100
- ID_WINDOW_LEFT=101
- ID_WINDOW_RIGHT=102
- ID_WINDOW_BOTTOM=103
+ ID_WINDOW_TOP = 100
+ ID_WINDOW_LEFT = 101
+ ID_WINDOW_RIGHT = 102
+ ID_WINDOW_BOTTOM = 103
ID_CLOSE_COMPOSITE_WORKER = wx.NewId()
ID_UPDATE_PYPHANT = wx.NewId()
def __init__(self, _wxPyphantApp):
- wx.Frame.__init__(self, None, -1, "wxPyphant %s" % __version__, size=(640,480))
+ wx.Frame.__init__(self, None, -1, "wxPyphant %s" % __version__,
+ size=(640,480))
import PyphantCanvas
- self._statusBar=self.CreateStatusBar()
- self._wxPyphantApp=_wxPyphantApp
+ self._statusBar = self.CreateStatusBar()
+ self._wxPyphantApp = _wxPyphantApp
self._initMenuBar()
self._initSash()
- self.recipeState=None
+ self.recipeState = None
self.onOpenCompositeWorker(None)
self._workerRepository.Bind(wx.EVT_SASH_DRAGGED_RANGE,
self.onFoldPanelBarDrag,
id=self.ID_WINDOW_TOP,
id2=self.ID_WINDOW_BOTTOM)
self.Bind(wx.EVT_SIZE, self.onSize)
- self.compositeWorkerStack=[]
+ self.compositeWorkerStack = []
wx.MessageBox("Located log directory at %s.\n"
"Logging will go to %s." %
(LOGDIR, os.path.join(LOGDIR, 'pyphant.log')),
"Logging info")
def _initSash(self):
- self._workerRepository = wx.SashLayoutWindow(self,
- self.ID_WINDOW_RIGHT,
- wx.DefaultPosition,
- wx.Size(200,1000),
- wx.NO_BORDER |wx.SW_3D
- | wx.CLIP_CHILDREN)
+ self._workerRepository = wx.SashLayoutWindow(
+ self, self.ID_WINDOW_RIGHT, wx.DefaultPosition, wx.Size(200,1000),
+ wx.NO_BORDER | wx.SW_3D | wx.CLIP_CHILDREN)
self._workerRepository.SetDefaultSize(wx.Size(220,1000))
self._workerRepository.SetOrientation(wx.LAYOUT_VERTICAL)
self._workerRepository.SetAlignment(wx.LAYOUT_RIGHT)
@@ -159,7 +161,8 @@
if event.GetDragStatus() == wx.SASH_STATUS_OUT_OF_RANGE:
return
if event.GetId() == self.ID_WINDOW_RIGHT:
- self._workerRepository.SetDefaultSize(wx.Size(event.GetDragRect().width, 1000))
+ self._workerRepository.SetDefaultSize(
+ wx.Size(event.GetDragRect().width, 1000))
# Leaves bits of itself behind sometimes
wx.LayoutAlgorithm().LayoutWindow(self, self._remainingSpace)
self._remainingSpace.Refresh()
@@ -168,58 +171,58 @@
def onOpenCompositeWorker(self, event):
if not self._wxPyphantApp.pathToRecipe:
if pltform == 'Linux' or pltform == 'Darwin':
- osMessage = "Choose an existing recipe or cancel to create a new recipe"
- elif pltform=='Windows':
- osMessage = "Choose existing recipe to open or name a new recipe to create"
+ osMessage = "Choose an existing recipe or cancel to create "\
+ "a new recipe"
+ elif pltform == 'Windows':
+ osMessage = "Choose existing recipe to open or name a new "\
+ "recipe to create"
else:
raise OSError, "Operating System %s not supported!" % pltform
wc = "Pyphant Recipe(*.h5)|*.h5"
- dlg = wx.FileDialog( self, message=osMessage, defaultDir=os.getcwd(),
- defaultFile="", wildcard=wc, style=wx.OPEN)
+ dlg = wx.FileDialog(self, message=osMessage, defaultDir=os.getcwd(),
+ defaultFile="", wildcard=wc, style=wx.OPEN)
if dlg.ShowModal() == wx.ID_OK:
self._wxPyphantApp.pathToRecipe = dlg.GetPath()
else:
dlg.Destroy()
- dlg = wx.FileDialog( self, message='Create a new recipe', defaultDir=os.getcwd(),
- defaultFile="", wildcard=wc, style=wx.SAVE)
+ dlg = wx.FileDialog(self, message='Create a new recipe',
+ defaultDir=os.getcwd(), defaultFile="",
+ wildcard=wc, style=wx.SAVE)
if dlg.ShowModal() == wx.ID_OK:
path = dlg.GetPath()
- if not path[:-3]=='.h5':
- path+='.h5'
+ if not path[:-3] == '.h5':
+ path += '.h5'
self._wxPyphantApp.pathToRecipe = path
dlg.Destroy()
-
import PyphantCanvas
if self._wxPyphantApp.pathToRecipe[-3:] == '.h5':
if os.path.exists(self._wxPyphantApp.pathToRecipe):
- recipe = pyphant.core.PyTablesPersister.loadRecipeFromHDF5File(self._wxPyphantApp.pathToRecipe)
- #from pyphant.core import KnowledgeManager
- #KnowledgeManager.KnowledgeManager.getInstance().registerURL(
- # "file:///"+os.path.realpath(self._wxPyphantApp.pathToRecipe)
- # )
- self._remainingSpace=PyphantCanvas.PyphantCanvas(self, recipe)
+ recipe = pyphant.core.PyTablesPersister.loadRecipeFromHDF5File(
+ self._wxPyphantApp.pathToRecipe)
+ self._remainingSpace = PyphantCanvas.PyphantCanvas(self, recipe)
else:
- self._remainingSpace=PyphantCanvas.PyphantCanvas(self)
+ self._remainingSpace = PyphantCanvas.PyphantCanvas(self)
else:
- raise IOError("Unknown file format in file \""+self._wxPyphantApp.pathToRecipe+"\"")
- self.recipeState='clean'
+ raise IOError("Unknown file format in file \"%\""\
+ % self._wxPyphantApp.pathToRecipe)
+ self.recipeState = 'clean'
self._remainingSpace.diagram.recipe.registerListener(self.recipeChanged)
def recipeChanged(self, event):
- self.recipeState='dirty'
+ self.recipeState = 'dirty'
def onSaveCompositeWorker(self, event=None):
pyphant.core.PyTablesPersister.saveRecipeToHDF5File(
self._remainingSpace.diagram.recipe,
self._wxPyphantApp.pathToRecipe,
self._fileMenu.IsChecked(wx.ID_FILE4))
- self.recipeState='clean'
+ self.recipeState = 'clean'
def onSaveAsCompositeWorker(self, event=None):
msg = "Select file to save recipe."
wc = "Pyphant recipe (*.h5)|*.h5"
- dlg = wx.FileDialog(self, message = msg, defaultDir = os.getcwd(),
- defaultFile = "", wildcard = wc, style = wx.SAVE)
+ dlg = wx.FileDialog(self, message=msg, defaultDir=os.getcwd(),
+ defaultFile="", wildcard=wc, style=wx.SAVE)
if dlg.ShowModal() == wx.ID_OK:
filename = dlg.GetPath()
if not filename.endswith(".h5"):
@@ -229,30 +232,32 @@
filename,
self._fileMenu.IsChecked(wx.ID_FILE4))
self._wxPyphantApp.pathToRecipe = filename
- self.recipeState='clean'
+ self.recipeState = 'clean'
else:
dlg.Destroy()
def _initMenuBar(self):
self._menuBar = wx.MenuBar()
self._fileMenu = wx.Menu()
- #self._fileMenu.Append( wx.ID_NEW, "&New\tCTRL+n")
- #self._fileMenu.Append( wx.ID_OPEN, "&Open\tCTRL+o")
+ #self._fileMenu.Append(wx.ID_NEW, "&New\tCTRL+n")
+ #self._fileMenu.Append(wx.ID_OPEN, "&Open\tCTRL+o")
self._fileMenu.AppendCheckItem(wx.ID_FILE4, "Save &results\tCTRL+r")
self._fileMenu.Check(wx.ID_FILE4, True)
- self._fileMenu.Append( wx.ID_SAVE, "&Save\tCTRL+s")
- self._fileMenu.Append( wx.ID_SAVEAS, "Save &as\tCTRL+a")
- self._fileMenu.Append( wx.ID_EXIT, "E&xit" )
- self._fileMenu.Append( wx.ID_FILE1, "Import HDF5 or FMF from &URL" )
- self._fileMenu.Append( wx.ID_FILE2, "&Import local HDF5 or FMF file")
- self._fileMenu.Append( wx.ID_FILE3, "Start/pause sharing &knowledge")
- self._menuBar.Append( self._fileMenu, "&File" )
+ self._fileMenu.Append(wx.ID_SAVE, "&Save\tCTRL+s")
+ self._fileMenu.Append(wx.ID_SAVEAS, "Save &as\tCTRL+a")
+ self._fileMenu.Append(wx.ID_EXIT, "E&xit")
+ self._fileMenu.Append(wx.ID_FILE1, "Import HDF5 or FMF from &URL")
+ self._fileMenu.Append(wx.ID_FILE2, "&Import local HDF5 or FMF file")
+ self._fileMenu.Append(wx.ID_FILE3, "Start/pause sharing &knowledge")
+ self._menuBar.Append(self._fileMenu, "&File")
self._closeCompositeWorker = wx.Menu()
- self._closeCompositeWorker.Append(self.ID_CLOSE_COMPOSITE_WORKER, "&Close Composite Worker")
- self._menuBar.Append( self._closeCompositeWorker, "&Close Composite Worker")
+ self._closeCompositeWorker.Append(self.ID_CLOSE_COMPOSITE_WORKER,
+ "&Close Composite Worker")
+ self._menuBar.Append(self._closeCompositeWorker,
+ "&Close Composite Worker")
self._updateMenu = self.createUpdateMenu()
- self._menuBar.Append( self._updateMenu, "&Update")
- self.SetMenuBar( self._menuBar )
+ self._menuBar.Append(self._updateMenu, "&Update")
+ self.SetMenuBar(self._menuBar)
self._menuBar.EnableTop(1, False)
#self.Bind(wx.EVT_MENU, self.onCreateNew, id=wx.ID_NEW)
#self.Bind(wx.EVT_MENU, self.onOpenCompositeWorker, id=wx.ID_OPEN)
@@ -260,21 +265,23 @@
self.Bind(wx.EVT_MENU, self.onSaveAsCompositeWorker, id=wx.ID_SAVEAS)
self.Bind(wx.EVT_CLOSE, self.onClose)
self.Bind(wx.EVT_MENU, self.onQuit, id=wx.ID_EXIT)
- self.Bind(wx.EVT_MENU, self.onCloseCompositeWorker, id=self.ID_CLOSE_COMPOSITE_WORKER)
+ self.Bind(wx.EVT_MENU, self.onCloseCompositeWorker,
+ id=self.ID_CLOSE_COMPOSITE_WORKER)
self.Bind(wx.EVT_MENU, self.onImportURL, id=wx.ID_FILE1)
self.Bind(wx.EVT_MENU, self.onImportLocal, id=wx.ID_FILE2)
self.Bind(wx.EVT_MENU, self.onShare, id=wx.ID_FILE3)
def createUpdateMenu(self):
updateMenu = wx.Menu()
- updateMenu.Append( self.ID_UPDATE_PYPHANT, "Update &Pyphant" )
+ updateMenu.Append(self.ID_UPDATE_PYPHANT, "Update &Pyphant")
self.Bind(wx.EVT_MENU, self.onUpdatePyphant, id=self.ID_UPDATE_PYPHANT)
self.updateIds = { self.ID_UPDATE_PYPHANT : 'pyphant' }
for toolbox in pkg_resources.iter_entry_points("pyphant.workers"):
dist = toolbox.dist
nId = wx.NewId()
self.updateIds[nId] = dist.key
- updateMenu.Append( nId, "Update %s (%s)" % (dist.project_name, dist.version) )
+ updateMenu.Append(nId, "Update %s (%s)"\
+ % (dist.project_name, dist.version))
self.Bind(wx.EVT_MENU, self.onUpdatePyphant, id=nId)
return updateMenu
@@ -286,11 +293,13 @@
self.Close()
def onClose(self, event):
- dlgid=None
- if self.recipeState!='clean':
+ dlgid = None
+ if self.recipeState != 'clean':
cpt = "Save changed recipe?"
- msg = "The recipe has changed since the last saving.\nDo you want to save before terminating?"
- dlg = wx.MessageDialog(self, msg, cpt, style=wx.YES|wx.NO|wx.CANCEL|wx.ICON_QUESTION)
+ msg = "The recipe has changed since the last saving.\n"\
+ "Do you want to save before terminating?"
+ dlg = wx.MessageDialog(
+ self, msg, cpt, style=wx.YES|wx.NO|wx.CANCEL|wx.ICON_QUESTION)
dlgid = dlg.ShowModal()
if dlgid == wx.ID_YES:
self.onSaveCompositeWorker()
@@ -306,24 +315,25 @@
def editCompositeWorker(self, worker):
import PyphantCanvas
self.compositeWorkerStack.append(self._remainingSpace)
- self._remainingSpace=PyphantCanvas.PyphantCanvas(self, worker)
+ self._remainingSpace = PyphantCanvas.PyphantCanvas(self, worker)
self._remainingSpace.diagram.recipe.registerListener(self.recipeChanged)
self._menuBar.EnableTop(1, True)
def onCloseCompositeWorker(self, event):
self._remainingSpace.Destroy()
- self._remainingSpace=self.compositeWorkerStack.pop()
- if len(self.compositeWorkerStack)==0:
+ self._remainingSpace = self.compositeWorkerStack.pop()
+ if len(self.compositeWorkerStack) == 0:
self._menuBar.EnableTop(1, False)
def onImportURL(self, event):
cpt = "Import HDF5 or FMF from URL"
- msg = "Enter an URL to a valid HDF5 or FMF file \
-(e.g. http://www.example.org/data.h5).\n\
-The file is stored permanently in your home directory in the \
-.pyphant directory\nand all DataContainers contained in that file are \
-available by using the\nEmd5Src Worker even after restarting wxPyphant.\n\
-HTTP redirects are resolved automatically, i.e. DOIs are supported as well."
+ msg = "Enter an URL to a valid HDF5 or FMF file "\
+ "(e.g. http://www.example.org/data.h5).\n"\
+ "The file is stored permanently in your home directory in the "\
+ ".pyphant directory\nand all DataContainers contained in that "\
+ "file are available by using the\nEmd5Src Worker even after "\
+ "restarting wxPyphant.\nHTTP redirects are resolved "\
+ "automatically, i.e. DOIs are supported as well."
dlg = wx.TextEntryDialog(self, msg, cpt)
dlgid = dlg.ShowModal()
if dlgid != wx.ID_CANCEL:
@@ -345,8 +355,8 @@
def onImportLocal(self, event):
msg = "Select HDF5 or FMF file to import DataContainer(s) from."
wc = "*.h5, *.hdf, *.hdf5, *.fmf|*.h5;*.hdf;*.hdf5;*.fmf"
- dlg = wx.FileDialog(self, message = msg, defaultDir = os.getcwd(),
- defaultFile = "", wildcard = wc, style = wx.OPEN)
+ dlg = wx.FileDialog(self, message=msg, defaultDir=os.getcwd(),
+ defaultFile="", wildcard=wc, style=wx.OPEN)
if dlg.ShowModal() == wx.ID_OK:
filename = dlg.GetPath()
url = 'file://' + os.path.realpath(filename)
@@ -358,8 +368,8 @@
km.registerURL(url)
except Exception:
cpt2 = "Error"
- msg2 = "'%s' is not a valid HDF5 or FMF file.\n\
-(Tried to import from '%s')" % (filename, url)
+ msg2 = "'%s' is not a valid HDF5 or FMF file.\n"\
+ "(Tried to import from '%s')" % (filename, url)
finally:
dlg2 = wx.MessageDialog(self, msg2, cpt2, wx.OK)
dlgid2 = dlg2.ShowModal()
Modified: trunk/src/workers/tools/tools/__init__.py
===================================================================
--- trunk/src/workers/tools/tools/__init__.py 2010-02-25 17:31:27 UTC (rev 670)
+++ trunk/src/workers/tools/tools/__init__.py 2010-03-14 14:26:07 UTC (rev 671)
@@ -40,10 +40,9 @@
# $Source$
workers=[
- "Emd5Src",
- "BatchHead",
- "BatchTail",
- "BatchExtractor",
- "ParameterRun"
+ "Emd5Src"#,
+# "BatchHead",
+# "BatchTail",
+# "BatchExtractor",
+# "ParameterRun"
]
-
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-25 17:31:33
|
Revision: 670
http://pyphant.svn.sourceforge.net/pyphant/?rev=670&view=rev
Author: zklaus
Date: 2010-02-25 17:31:27 +0000 (Thu, 25 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Adapt NonUniformImage to newer matplotlib.
Modified Paths:
--------------
trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py
Modified: trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py
===================================================================
--- trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py 2010-02-22 18:20:31 UTC (rev 669)
+++ trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py 2010-02-25 17:31:27 UTC (rev 670)
@@ -134,14 +134,15 @@
pylab.colorbar(format=F(self.fieldContainer), ax=ax)
else:
im = NonUniformImage(ax, extent=(xmin,xmax,ymin,ymax))
- im.set_data(x, y, self.fieldContainer.maskedData)
- ax.images.append(im)
- ax.set_xlim(xmin,xmax)
- ax.set_ylim(ymin,ymax)
if vmin is not None or vmax is not None:
im.set_clim(vmin, vmax)
+ im.set_data(x, y, self.fieldContainer.maskedData)
else:
+ im.set_data(x, y, self.fieldContainer.maskedData)
im.autoscale_None()
+ ax.images.append(im)
+ ax.set_xlim(xmin,xmax)
+ ax.set_ylim(ymin,ymax)
pylab.colorbar(im,format=F(self.fieldContainer), ax=ax)
pylab.xlabel(self.fieldContainer.dimensions[-1].shortlabel)
pylab.ylabel(self.fieldContainer.dimensions[-2].shortlabel)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-22 18:20:38
|
Revision: 669
http://pyphant.svn.sourceforge.net/pyphant/?rev=669&view=rev
Author: zklaus
Date: 2010-02-22 18:20:31 +0000 (Mon, 22 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Workaround: Fix stupid numpy types for parameters on load.
Enh: Better handling of refreshParam in recipe loading.
Fix: Import Quantity as PhysicalQuantity for backwards compatibility.
Enh: Cross thread exception handling via queues.
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/Connectors.py
trunk/src/pyphant/pyphant/core/PyTablesPersister.py
trunk/src/pyphant/pyphant/wxgui2/ConfigureFrame.py
trunk/src/pyphant/pyphant/wxgui2/PyphantDiagram.py
Modified: trunk/src/pyphant/pyphant/core/Connectors.py
===================================================================
--- trunk/src/pyphant/pyphant/core/Connectors.py 2010-02-21 20:51:58 UTC (rev 668)
+++ trunk/src/pyphant/pyphant/core/Connectors.py 2010-02-22 18:20:31 UTC (rev 669)
@@ -38,6 +38,7 @@
# $Source$
import copy, threading, inspect, logging
+import Queue
class FullSocketError(ValueError):
pass
@@ -71,20 +72,22 @@
self.worker.id+"."+self.pre+self.name.capitalize())
class Computer(threading.Thread):
- def __init__(self, method, **kwargs):
+ def __init__(self, method, exception_queue, **kwargs):
threading.Thread.__init__(self)
self.method = method
self.kwargs = kwargs
self.result = None
+ self.exception_queue = exception_queue
def run(self):
if self.method:
try:
self.result = self.method(subscriber=self.kwargs["subscriber"])
- except:
+ except Exception, e:
logging.getLogger('pyphant').debug(
u"An unhandled exception occured in the calculation.",
exc_info = True)
+ self.exception_queue.put(e)
raise
@@ -148,13 +151,20 @@
sockets = args[1:-1]
name = method.func_name+'PyphantWrapper'
l = 'def '+name+'(subscriber, method=method, process=self):\n'
+ l += '\texception_queue=Queue.Queue()\n'
for s in sockets:
l += '\t'+s+'=Computer(method.im_self.getSocket("'
- l += s+'").getResult, subscriber=subscriber)\n'
+ l += s+'").getResult, exception_queue, subscriber=subscriber)\n'
for s in sockets:
l += '\t'+s+'.start()\n'
for s in sockets:
l += '\t'+s+'.join()\n'
+ l += '\texceptions=[]\n'
+ l += '\twhile not exception_queue.empty():\n'
+ l += '\t\texceptions.append(exception_queue.get())\n'
+ l += '\t\texception_queue.task_done()\n'
+ l += '\tif len(exceptions)>0:\n'
+ l += '\t\traise RuntimeError, str(exceptions)\n'
#If no sockets are needed the comma in the next line will be erased,
#so do not add a space!
l += '\treturn method( subscriber=Updater(subscriber, process),'
@@ -187,7 +197,6 @@
self._resultLock.release()
return result
-
class Socket(Connector):
def __init__(self, worker, name, type=DEFAULT_DATA_TYPE):
Connector.__init__(self, worker, name, type, "socket")
Modified: trunk/src/pyphant/pyphant/core/PyTablesPersister.py
===================================================================
--- trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-02-21 20:51:58 UTC (rev 668)
+++ trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-02-22 18:20:31 UTC (rev 669)
@@ -67,6 +67,7 @@
from pyphant.core import (CompositeWorker, DataContainer)
from tables import StringCol, Col
from pyphant.quantities import Quantity
+from pyphant.quantities import Quantity as PhysicalQuantity
from ImageProcessing.AutoFocus import FocusSlice # For loading FCs...
import scipy
import logging
@@ -240,10 +241,23 @@
def restoreParamsToWorkers(recipeGroup, workers):
for workerGroup in recipeGroup:
worker = workers[workerGroup._v_name]
- worker.refreshParams()
+ try:
+ worker.refreshParams()
+ except:
+ _logger.warning(u"Attempted refreshParam failed for %s. Check Parameters!"%worker.name,
+ exc_info = True)
for paramName in workerGroup.parameters._v_attrs._v_attrnamesuser:
param = getattr(workerGroup.parameters._v_attrs, paramName)
- worker.getParam(paramName).overrideValue(param)
+ if type(param)==scipy.ndarray:
+ param=unicode(param)
+ elif type(param)==scipy.string_:
+ param=str(param)
+ elif type(param)==scipy.int32:
+ param=int(param)
+ try:
+ worker.getParam(paramName).overrideValue(param)
+ except KeyError:
+ _logger.warning(u'Could not restore "%s" to parameter: "%s"'%(param,paramName))
def loadRecipeFromHDF5File( filename ):
h5 = tables.openFile(filename, 'r')
Modified: trunk/src/pyphant/pyphant/wxgui2/ConfigureFrame.py
===================================================================
--- trunk/src/pyphant/pyphant/wxgui2/ConfigureFrame.py 2010-02-21 20:51:58 UTC (rev 668)
+++ trunk/src/pyphant/pyphant/wxgui2/ConfigureFrame.py 2010-02-22 18:20:31 UTC (rev 669)
@@ -37,6 +37,7 @@
__version__ = "$Revision$"
# $Source$
+import Queue
import wx
from pyphant.core.Connectors import Computer
@@ -49,7 +50,8 @@
wx.Dialog.__init__(self, parent, -1, "Configure "+worker.getParam("name").value)
self._paramDict={}
progress = ProgressMeter("Acquiring Param data")
- computer = Computer(worker.refreshParams, subscriber=progress)
+ exception_queue = Queue.Queue()
+ computer = Computer(worker.refreshParams, exception_queue, subscriber=progress)
computer.start()
while (progress.percentage<100) and (computer.isAlive()):
progress.update()
Modified: trunk/src/pyphant/pyphant/wxgui2/PyphantDiagram.py
===================================================================
--- trunk/src/pyphant/pyphant/wxgui2/PyphantDiagram.py 2010-02-21 20:51:58 UTC (rev 668)
+++ trunk/src/pyphant/pyphant/wxgui2/PyphantDiagram.py 2010-02-22 18:20:31 UTC (rev 669)
@@ -38,6 +38,7 @@
# $Source$
import threading, math
+import Queue
import wx
import sogl
import pyphant.core.CompositeWorker
@@ -254,7 +255,8 @@
def visualize(self, event):
if not self._plug.resultIsAvailable():
progress = ProgressMeter(self._plug.name)
- computer = Connectors.Computer(self._plug.getResult, subscriber=progress)
+ exception_queue = Queue.Queue()
+ computer = Connectors.Computer(self._plug.getResult, exception_queue, subscriber=progress)
computer.start()
while (progress.percentage<100) and (computer.isAlive()):
progress.update()
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-21 20:52:06
|
Revision: 668
http://pyphant.svn.sourceforge.net/pyphant/?rev=668&view=rev
Author: zklaus
Date: 2010-02-21 20:51:58 +0000 (Sun, 21 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Module SQLiteWrapper adapted to additional base unit bit.
Conflicts resolved, but merged version still has bugs.
Refacturing: Removed intermediate module PhysicalQuantities, such that class Quantity is directly loaded from module quantities.
Refactoring: Renamed Function isPhysicalUnit to isUnit.
Refacturing: Renamed class PhysicalQuantity to Quantity.
Enh.: Added scenario for testing datetime with uncertainty.
Enh: Demo script readfmf.py accepts command 'parse <FMF-value>'.
Refactured: Function item2value takes a simple string as argument and not a pair of dictionary with associated key.
Refactured: Function item2value takes a simple string as argument and not a pair of dictionary with associated key.
Bugfix: File example.fmf does not specify complex numbers with capital J anymore.
Bugfix: Items are correctly discriminated between complex numbers and quantities of unit Joule.
Defined 1Byte=8Bit.
Enh.: Added binary Units bit and B.
Test for converting value 1EUR to 1USD is only run, if up to date exchange rates have been fetched.
Enh: Added binary units.
Bugfix: Monetary values cannot have prefixes with powers lower than 1000.
Enh.: If fields cannot be reshaped, because they consists of strings. A warning is given.
Modified Paths:
--------------
trunk/doc/demo/example.fmf
trunk/doc/demo/readfmf.py
trunk/src/pyphant/pyphant/core/DataContainer.py
trunk/src/pyphant/pyphant/core/FieldContainer.py
trunk/src/pyphant/pyphant/core/H5FileHandler.py
trunk/src/pyphant/pyphant/core/KnowledgeManager.py
trunk/src/pyphant/pyphant/core/PyTablesPersister.py
trunk/src/pyphant/pyphant/core/SQLiteWrapper.py
trunk/src/pyphant/pyphant/quantities/ParseQuantities.py
trunk/src/pyphant/pyphant/quantities/__init__.py
trunk/src/pyphant/pyphant/tests/TestAutoFocus.py
trunk/src/pyphant/pyphant/tests/TestDataContainer.py
trunk/src/pyphant/pyphant/tests/TestH5FileHandler.py
trunk/src/pyphant/pyphant/tests/TestParseQuantities.py
trunk/src/pyphant/pyphant/tests/TestPyTablesPersister.py
trunk/src/pyphant/pyphant/tests/TestSQLiteWrapper.py
trunk/src/pyphant/pyphant/visualizers/Chart.py
trunk/src/pyphant/pyphant/visualizers/External.py
trunk/src/pyphant/pyphant/visualizers/ImageVisualizer.py
trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py
trunk/src/workers/ImageProcessing/ImageProcessing/DistanceMapper.py
trunk/src/workers/ImageProcessing/ImageProcessing/ImageLoaderWorker.py
trunk/src/workers/ImageProcessing/ImageProcessing/NDImageWorker.py
trunk/src/workers/ImageProcessing/ImageProcessing/SlopeCalculator.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestApplyMask.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestDistanceMapper.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestImageLoader.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestMedianiser.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestSkeletonizeFeature.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestSlopeCalculator.py
trunk/src/workers/ImageProcessing/ImageProcessing/tests/TestThresholdingWorker.py
trunk/src/workers/OSC/OSC/CompareFields.py
trunk/src/workers/OSC/OSC/ComputeFunctional.py
trunk/src/workers/OSC/OSC/ErrorEstimator.py
trunk/src/workers/OSC/OSC/EstimateParameter.py
trunk/src/workers/OSC/OSC/ExtremumFinder.py
trunk/src/workers/OSC/OSC/MRA.py
trunk/src/workers/OSC/OSC/OscAbsorption.py
trunk/src/workers/OSC/OSC/OscCurrent.py
trunk/src/workers/OSC/OSC/OscLoader.py
trunk/src/workers/OSC/OSC/OscVisualisers.py
trunk/src/workers/OSC/OSC/Slicing.py
trunk/src/workers/OSC/OSC/Smoother.py
trunk/src/workers/OSC/OSC/ThicknessModeller.py
trunk/src/workers/OSC/OSC/tests/TestExtremumFinder.py
trunk/src/workers/OSC/OSC/tests/TestMRA.py
trunk/src/workers/OSC/OSC/tests/TestOscAbsorption.py
trunk/src/workers/Statistics/Statistics/tests/TestHistogram.py
trunk/src/workers/fmfile/fmfile/FMFLoader.py
trunk/src/workers/fmfile/fmfile/tests/TestFMFLoader.py
trunk/src/workers/tools/tools/ParameterRun.py
trunk/src/workers/tools/tools/tests/TestEmd5Source.py
Added Paths:
-----------
trunk/src/pyphant/pyphant/tests/TestQuantities.py
Removed Paths:
-------------
trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py
Property Changed:
----------------
trunk/doc/demo/readfmf.py
Modified: trunk/doc/demo/example.fmf
===================================================================
--- trunk/doc/demo/example.fmf 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/doc/demo/example.fmf 2010-02-21 20:51:58 UTC (rev 668)
@@ -28,10 +28,10 @@
float with Exponential: -1.1E10
list of floats: 1.0, .1, 1e-10, -1.1E10
complex: 1+2j
-Complex: 1+2J
-Complex number with zero real part : 2J
-Complex number with zero imaginary part: 1+0J
-list of complex: 1+2j, 1+2J
+Complex: 1.1+2j
+Complex number with zero real part : 2j
+Complex number with zero imaginary part: 1+0j
+list of complex: 1+2j, 1.1+2j
[Boolean values]
true1: True
true2: true
@@ -101,7 +101,7 @@
[*data: T]
H_2 1 1. 1e1 1+0j nan inf
O_2 2 .2 2E1 2+.1j NaN INF
-O 2 2 .2 2E1 2+.1J NAN Inf
+O 2 2 .2 2E1 2.+2j NAN Inf
[*data definitions: M]
String: S
Complex: C
Modified: trunk/doc/demo/readfmf.py
===================================================================
--- trunk/doc/demo/readfmf.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/doc/demo/readfmf.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,15 +1,61 @@
-import sys
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# * Neither the name of the Freiburg Materials Research Center,
+# University of Freiburg nor the names of its contributors may be used to
+# endorse or promote products derived from this software without specific
+# prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
+# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
import logging
+from optparse import OptionParser
+
logging.basicConfig(level=logging.WARNING,
format="%(asctime)s - %(levelname)s:%(name)s:%(thread)d:%(module)s.%(funcName)s(l %(lineno)d):%(message)s")
-from fmfile.FMFLoader import FMFLoader
+from fmfile.FMFLoader import FMFLoader,item2value
-worker = FMFLoader()
-if len(sys.argv) == 1:
+parser = OptionParser(usage="""usage: %prog path2FMFfile
+ %prog parse FMFvalue""")
+
+(options, args) = parser.parse_args()
+if len(args) == 0:
filenames = ['example.fmf']
+elif args[0] in ('parse',u'parse'):
+ value = ' '.join(args[1:])
+ result = item2value(value)
+ print "Value \"%s\" is interpreted as %s:" % (value,type(result))
+ print result
+ from sys import exit
+ exit(0)
else:
- filenames = sys.argv[1:]
+ filenames = args
+
+worker = FMFLoader()
for filename in filenames:
worker.paramFilename.value=filename
result = worker.plugLoadFMF.getResult()
Property changes on: trunk/doc/demo/readfmf.py
___________________________________________________________________
Added: svn:executable
+ *
Modified: trunk/src/pyphant/pyphant/core/DataContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/DataContainer.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/DataContainer.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -70,8 +71,8 @@
import copy, hashlib, threading, numpy, StringIO
import os, platform, datetime, socket, urlparse
-from pyphant.quantities.PhysicalQuantities import (isPhysicalQuantity,
- PhysicalQuantity)
+from pyphant.quantities import (isQuantity,
+ Quantity)
import Helpers
import logging
@@ -195,7 +196,7 @@
\t .data \t- Table of samples stored in a numpy.ndarray.
\t .desc \t- Description numpy.dtype of the ndarray.
-\t .units \t- List of PhysicalQuantities objects denoting the units of
+\t .units \t- List of quantities.objects denoting the units of
\t\t\t the columns.
\t .longname \t- Notation of the data, e.g. 'database query',
\t\t\t which is used for the automatic annotation of charts.
@@ -338,12 +339,12 @@
al.append(('SCColumn', column))
else:
try:
- phq = PhysicalQuantity(e)
+ phq = Quantity(e)
al.append(('PhysQuant', phq))
continue
except:
try:
- number = PhysicalQuantity(e+' m')
+ number = Quantity(e+' m')
al.append(('Number', eval(e)))
continue
except: pass
@@ -594,7 +595,7 @@
<atomar> := <value> <CompareOp> <value>
where <value> is either a SC Column accessed as
"longname" or "shortname" (including the double quotes)
- or a number or a string representing a PhysicalQuantity
+ or a number or a string representing a Quantity
(e.g. 300nm). And <CompareOp> can be ==, !=, <, <=, >, >=.
Then a valid expression <expression> is:
- <atomar>
@@ -613,7 +614,7 @@
- or
If expression is a nested tuple, syntax is as follows:
<value> is either ('SCColumn', FieldContainer instance)
- or ('PhysQuant', PhysicalQuantity instance)
+ or ('PhysQuant', Quantity instance)
or ('Number', int float etc.)
<CompareOp> is in ['==', '!=', ...]
A valid nested tuple <nt> can be:
@@ -661,7 +662,7 @@
rightvalue = None
if left[0] == 'SCColumn' and right[0] == 'SCColumn':
number = right[1].unit/left[1].unit
- if isPhysicalQuantity(number):
+ if isQuantity(number):
raise TypeError(
'Cannot compare "' + left[1].longname + '" to "'
+ right[1].longname + '".'
@@ -670,7 +671,7 @@
rightvalue = right[1].data*number
elif left[0] == 'SCColumn':
number = right[1]/left[1].unit
- if isPhysicalQuantity(number):
+ if isQuantity(number):
raise TypeError(
'Cannot compare "' + left[1].longname
+ '" to ' + str(right[1]) + '".'
@@ -679,7 +680,7 @@
rightvalue = number
elif right[0] == 'SCColumn':
number = left[1]/right[1].unit
- if isPhysicalQuantity(number):
+ if isQuantity(number):
raise TypeError(
"Cannot compare " + str(left[1]) + ' to "'
+ right[1].longname + '".'
Modified: trunk/src/pyphant/pyphant/core/FieldContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/FieldContainer.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/FieldContainer.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,7 +1,8 @@
# -*- coding: utf-8 -*-
from __future__ import with_statement
-# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -59,7 +60,7 @@
import scipy, copy, hashlib, threading, numpy, StringIO
import os, platform, datetime, socket, urlparse
-from pyphant.quantities.PhysicalQuantities import (isPhysicalQuantity, PhysicalQuantity,_prefixes)
+from pyphant.quantities import (isQuantity, Quantity,_prefixes)
from DataContainer import DataContainer, enc, _logger
from types import NoneType
@@ -136,14 +137,14 @@
def slice2ind(arg, dim):
if isinstance(arg, type("")):
- sl = [ "0"+a for a in arg.split(':')] #Hack for PhysicalQuantities, which does not recognize .6 as a number.
+ sl = [ "0"+a for a in arg.split(':')] #Hack for quantities. which does not recognize .6 as a number.
unit = dim.unit
try:
- hi = PhysicalQuantity(sl[1])
+ hi = Quantity(sl[1])
try:
- li = PhysicalQuantity(sl[0])
+ li = Quantity(sl[0])
except:
- li = PhysicalQuantity(float(sl[0]),hi.unit)
+ li = Quantity(float(sl[0]),hi.unit)
try:
li, hi = [ i.inUnitsOf(unit.unit).value/unit.value for i in [li,hi] ]
except TypeError:
@@ -182,7 +183,7 @@
\t\t\t shortname=u"\\Psi",rescale=False)
\t Class describing sampled fields:
\t .data\t\t- Numpy.array representing the sampled field.
-\t .unit\t\t- PhysicalQuantity object denoting the unit of the sampled field.
+\t .unit\t\t- Quantity object denoting the unit of the sampled field.
\t .dimensions\t- List of FieldContainer instances
\t\t\t describing the dimensions of the sampled field.
\t .data \t- Sampled field stored as numpy.array, which is rescaled to reasonable basic units if option rescale is chosen.
@@ -210,10 +211,10 @@
unit = unit.replace('^', '**')
if isinstance(unit, unicode):
unit = unit.encode('utf-8')
- self.unit = PhysicalQuantity(unit)
+ self.unit = Quantity(unit)
except:
try:
- self.unit = PhysicalQuantity("1"+unit)
+ self.unit = Quantity("1"+unit)
except:
self.unit = unit
self.error = error
@@ -244,7 +245,7 @@
dependency = ''
label = u"%s $%s%s$ / %s" % (self.longname.title(), self.shortname, dependency, self.unit)
try:
- if not isPhysicalQuantity(self.unit) and self.unit == 1:
+ if not isQuantity(self.unit) and self.unit == 1:
label = u"%s $%s%s$ / a.u." % (self.longname.title(),self.shortname,dependency)
except:
pass #just a ScientificPython bug
@@ -252,7 +253,7 @@
label=property(_getLabel)
def _getShortLabel(self):
- if not isPhysicalQuantity(self.unit) and self.unit == 1:
+ if not isQuantity(self.unit) and self.unit == 1:
if self.longname == 'index':
label = u"%s $%s$" % (self.longname.title(),self.shortname)
else:
@@ -311,11 +312,11 @@
super(FieldContainer, self).seal(id)
def inUnitsOf(self, other):
- if not isPhysicalQuantity(self.unit):
- if isPhysicalQuantity(other.unit):
+ if not isQuantity(self.unit):
+ if isQuantity(other.unit):
raise ValueError("Incompatible Units: self.unit = <%s>, other.unit = <%s>"%(self.unit, other.unit))
factor = float(self.unit)/float(other.unit)
- elif not isPhysicalQuantity(other.unit):
+ elif not isQuantity(other.unit):
raise ValueError("Incompatible Units: self.unit = <%s>, other.unit = <%s>"%(self.unit, other.unit))
else:
if not self.unit.isCompatible(other.unit.unit):
@@ -329,7 +330,7 @@
return newSelf
def rescale(self):
- if isPhysicalQuantity(self.unit):
+ if isQuantity(self.unit):
oldUnit = self.unit.inBaseUnits()
else:
return
@@ -391,7 +392,7 @@
error = self.error
otherData = other.data
otherError = other.error
- if (isPhysicalQuantity(self.unit) or isPhysicalQuantity(other.unit)):
+ if (isQuantity(self.unit) or isQuantity(other.unit)):
try:
if not (self.unit.inBaseUnits().unit == other.unit.inBaseUnits().unit):
_logger.debug('The units are different.')
@@ -470,8 +471,8 @@
for i in xrange(len(self._dimensions)):
if not self._dimensions[i] == other.dimensions[i]:
return NotImplemented
- if isPhysicalQuantity(self.unit):
- if not isPhysicalQuantity(other.unit):
+ if isQuantity(self.unit):
+ if not isQuantity(other.unit):
return NotImplemented
if not self.unit.isCompatible(other.unit.unit):
return NotImplemented
@@ -481,7 +482,7 @@
else:
data = other.data+(self.data*self.unit.value*self.unit.unit.conversionFactorTo(other.unit.unit))/other.unit.value
unit = other.unit
- elif isPhysicalQuantity(other.unit):
+ elif isQuantity(other.unit):
return NotImplemented
else:
data = (self.data*self.unit) + (other.data*other.unit)
@@ -510,8 +511,8 @@
for i in xrange(len(self._dimensions)):
if not self._dimensions[i] == other.dimensions[i]:
return NotImplemented
- if isPhysicalQuantity(self.unit):
- if not (isPhysicalQuantity(other.unit) and self.unit.isCompatible(other.unit.unit)):
+ if isQuantity(self.unit):
+ if not (isQuantity(other.unit) and self.unit.isCompatible(other.unit.unit)):
return NotImplemented
if self.unit >= other.unit:
data = self.data - (other.data*other.unit.value*other.unit.unit.conversionFactorTo(self.unit.unit))/self.unit.value
@@ -520,7 +521,7 @@
data = ((self.data*self.unit.value*self.unit.unit.conversionFactorTo(other.unit.unit))/other.unit.value) - other.data
unit = other.unit
else:
- if isPhysicalQuantity(other.unit):
+ if isQuantity(other.unit):
return NotImplemented
data = (self.data*self.unit) - (other.data*other.unit)
unit = 1.0
Modified: trunk/src/pyphant/pyphant/core/H5FileHandler.py
===================================================================
--- trunk/src/pyphant/pyphant/core/H5FileHandler.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/H5FileHandler.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2006-2010, Rectorate of the University of Freiburg
+# Copyright (c) 2009-2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -39,7 +40,7 @@
import tables
from pyphant.core import DataContainer
from tables import StringCol
-from pyphant.quantities.PhysicalQuantities import PhysicalQuantity
+from pyphant.quantities import Quantity
import scipy
import logging
import os
Modified: trunk/src/pyphant/pyphant/core/KnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -392,7 +392,7 @@
use (SQLiteWrapper instance).any_value
or (KM instance).any_value to skip value check
'storage': str types (==)
- 'unit': PhysicalUnit or number or PhysicalQuantity (==, FC only)
+ 'unit': PhysicalUnit or number or Quantity (==, FC only)
'dimensions': list of FC search dicts
(see above definitions, FC only)
'columns': list of FC search dicts (see above definitions, SC only)
@@ -410,7 +410,7 @@
--> [('name1', ), ('name2', ), ...]
Get id and shortname of all FCs that are parametrized by
a time dimension along the primary axis:
- tunit = PhysicalQuantity(1, 's')
+ tunit = Quantity(1, 's')
get_andsearch_result(['id', 'shortname'],
{'type':'field',
'dimensions':[{'unit':tunit}]})
Modified: trunk/src/pyphant/pyphant/core/PyTablesPersister.py
===================================================================
--- trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -65,7 +66,7 @@
import sys
from pyphant.core import (CompositeWorker, DataContainer)
from tables import StringCol, Col
-from pyphant.quantities.PhysicalQuantities import PhysicalQuantity
+from pyphant.quantities import Quantity
from ImageProcessing.AutoFocus import FocusSlice # For loading FCs...
import scipy
import logging
Modified: trunk/src/pyphant/pyphant/core/SQLiteWrapper.py
===================================================================
--- trunk/src/pyphant/pyphant/core/SQLiteWrapper.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/core/SQLiteWrapper.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2006-2010, Rectorate of the University of Freiburg
+# Copyright (c) 2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -41,17 +42,19 @@
import sqlite3
import time
from pyphant.core.Helpers import (utf82uc, uc2utf8, emd52dict)
-from pyphant.quantities.PhysicalQuantities import (PhysicalQuantity,
- PhysicalUnit)
+from pyphant.quantities import (Quantity,PhysicalUnit,_base_units)
from types import (FloatType, IntType, LongType, StringTypes)
def quantity2powers(quantity):
- if isinstance(quantity, PhysicalQuantity):
- return tuple(quantity.unit.powers)
+ numberOfBaseUnits = len(_base_units)
+ if isinstance(quantity, Quantity):
+ result = tuple(quantity.unit.powers)
+ assert len(result) == numberOfBaseUnits, "Expecting %i base units, but got a tupple of %i unit powers insteat." % (numberOfBaseUnits,len(result))
+ return result
elif isinstance(quantity, (FloatType, IntType, LongType)):
- return (0, ) * 10
+ return (0, ) * numberOfBaseUnits
else:
- raise ValueError("Expected (PhysicalQuantity, FloatType, IntType, "\
+ raise ValueError("Expected (Quantity, FloatType, IntType, "\
"LongType) but got %s instead."\
% (type(quantity), ))
@@ -68,11 +71,11 @@
def quantity2dbase(quantity):
if isinstance(quantity, (FloatType, IntType, LongType)):
return quantity.__repr__()
- elif isinstance(quantity, PhysicalQuantity):
+ elif isinstance(quantity, Quantity):
return "P%s;%s" % (quantity.value.__repr__(),
quantity.getUnitName())
else:
- raise ValueError("Expected (PhysicalQuantity, FloatType, IntType, "\
+ raise ValueError("Expected (Quantity, FloatType, IntType, "\
"LongType) but got %s instead."\
% (type(quantity), ))
@@ -80,7 +83,7 @@
if isinstance(dbase, StringTypes):
if dbase.startswith("P"):
tmp = dbase[1:].split(';')
- return PhysicalQuantity(str2number(tmp[0]), tmp[1])
+ return Quantity(str2number(tmp[0]), tmp[1])
else:
return str2number(dbase)
else:
@@ -273,7 +276,8 @@
('rad', 'INT'),
('sr', 'INT'),
('EUR', 'INT'),
- ('', 'UNIQUE(m, g, s, A, K, mol, cd, rad, sr, EUR)')]
+ ('bit', 'INT'),
+ ('', 'UNIQUE(m, g, s, A, K, mol, cd, rad, sr, EUR,bit)')]
create_table('km_base_units', columns, self.cursor)
#create triggers:
create_trigger('trigger_del_fc', 'DELETE', 'km_fc',
@@ -307,14 +311,14 @@
insert_dict['unit'] = quantity2dbase(summary['unit'])
try:
exe("INSERT OR ABORT INTO km_base_units "\
- "(m, g, s, A, K, mol, cd, rad, sr, EUR) "\
- "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+ "(m, g, s, A, K, mol, cd, rad, sr, EUR, bit) "\
+ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
quantity2powers(summary['unit']))
l_row_id = self.cursor.lastrowid
except sqlite3.IntegrityError:
exe("SELECT bu_id FROM km_base_units WHERE m=? AND g=? "\
"AND s=? AND A=? AND K=? AND mol=? AND cd=? AND rad=? "\
- "AND sr=? AND EUR=?", quantity2powers(summary['unit']))
+ "AND sr=? AND EUR=? AND bit=?", quantity2powers(summary['unit']))
tmp = self.cursor.fetchone()
assert tmp != None
l_row_id = tmp[0]
@@ -397,17 +401,17 @@
return key
def translate_unit_search(self, value):
- if isinstance(value, PhysicalQuantity):
+ if isinstance(value, Quantity):
value = value.unit.powers
elif isinstance(value, (IntType, LongType, FloatType)):
- value = [0] * 10
+ value = [0] * len(_base_units)
elif isinstance(value, PhysicalUnit):
value = value.powers
else:
raise ValueError(value)
expr = '(bu_id IN (SELECT bu_id FROM km_base_units WHERE '\
'm=? AND g=? AND s=? AND A=? AND K=? AND mol=? '\
- 'AND cd=? AND rad=? AND sr=? AND EUR=?))'
+ 'AND cd=? AND rad=? AND sr=? AND EUR=? AND bit=?))'
return (expr, value, True)
def translate_attr_search(self, value, type):
@@ -539,7 +543,7 @@
use (SQLiteWrapper instance).any_value
or (KM instance).any_value to skip value check
'storage': str types (==)
- 'unit': PhysicalUnit or number or PhysicalQuantity (==, FC only)
+ 'unit': PhysicalUnit or number or Quantity (==, FC only)
'dimensions': list of FC search dicts
(see above definitions, FC only)
'dim_of': str types: emd5 of parent FC (==, FC only)
@@ -559,7 +563,7 @@
--> [('name1', ), ('name2', ), ...]
Get id and shortname of all FCs that are parametrized by
a time dimension along the primary axis:
- tunit = PhysicalQuantity(1, 's')
+ tunit = Quantity(1, 's')
get_andsearch_result(['id', 'shortname'],
{'type':'field',
'dimensions':[{'unit':tunit}]})
Modified: trunk/src/pyphant/pyphant/quantities/ParseQuantities.py
===================================================================
--- trunk/src/pyphant/pyphant/quantities/ParseQuantities.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/quantities/ParseQuantities.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
-# Copyright (c) 2008, Rectorate of the University of Freiburg
+# Copyright (c) 2008-2009, Rectorate of the University of Freiburg
+# Copyright (c) 2009-2010, Andreas W. Liehr (li...@us...)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
@@ -37,10 +38,12 @@
__version__ = "$Revision$"
# $Source$
-from PhysicalQuantities import PhysicalQuantity
import mx.DateTime.ISO
+from pyphant.quantities import Quantity
def str2unit(unit):
+ """The function str2unit returns either a quantity or a float from a given string."""
+ # Prepare conversion to quantity
if unit.startswith('.'):
unit = '0'+unit
elif unit.endswith('%'):
@@ -55,9 +58,10 @@
unit = 1.0
elif not (unit[0].isdigit() or unit[0]=='-'):
unit = '1'+unit
+ # Convert input to quantity or float
try:
unit = unit.replace('^', '**')
- unit = PhysicalQuantity(unit.encode('utf-8'))
+ unit = Quantity(unit.encode('utf-8'))
except:
unit = float(unit)
return unit
@@ -94,9 +98,9 @@
def parseDateTime(value):
"""
>>>parseDateTime('2004-08-21 12:00:00+-12h')
- (PhysicalQuantity(731814.5,'d'), PhysicalQuantity(0.5,'d'))
+ (Quantity(731814.5,'d'), Quantity(0.5,'d'))
>>>parseDateTime('2004-08-21 12:00:00')
- (PhysicalQuantity(731814.5,'d'), None)
+ (Quantity(731814.5,'d'), None)
"""
datetimeWithError = value.split('+-')
if len(datetimeWithError)==2:
@@ -106,4 +110,4 @@
datetime = mx.DateTime.ISO.ParseAny(value)
error = None
days,seconds = datetime.absvalues()
- return (PhysicalQuantity(days,'d')+PhysicalQuantity(seconds,'s'),error)
+ return (Quantity(days,'d')+Quantity(seconds,'s'),error)
Deleted: trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py
===================================================================
--- trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py 2010-02-12 16:21:58 UTC (rev 667)
+++ trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py 2010-02-21 20:51:58 UTC (rev 668)
@@ -1,997 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright (c) 1998-2007, Konrad Hinsen <hi...@cn...>
-# Copyright (c) 2008, Rectorate of the University of Freiburg
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution.
-# * Neither the name of the Freiburg Materials Research Center,
-# University of Freiburg nor the names of its contributors may be used to
-# endorse or promote products derived from this software without specific
-# prior written permission.
-#
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
-# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
-# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
-# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
-# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
-# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""
-Quantities with units
-
-based on the module Scientific.Physics.PhysicalQuantities
-written by Conrad Hinsen with contributions from Greg Ward.
-
-This module provides a data type that represents a physical
-quantity together with its unit. It is possible to add and
-subtract these quantities if the units are compatible, and
-a quantity can be converted to another compatible unit.
-Multiplication, subtraction, and raising to integer powers
-is allowed without restriction, and the result will have
-the correct unit. A quantity can be raised to a non-integer
-power only if the result can be represented by integer powers
-of the base units.
-
-The values of physical constants are taken from the 1986
-recommended values from CODATA. Other conversion factors
-(e.g. for British units) come from various sources. I can't
-guarantee for the correctness of all entries in the unit
-table, so use this at your own risk.
-"""
-
-rc = { 'fetchCurrencyRates' : False }
-
-class NumberDict(dict):
-
- """
- Dictionary storing numerical values
-
- Constructor: NumberDict()
-
- An instance of this class acts like an array of number with
- generalized (non-integer) indices. A value of zero is assumed
- for undefined entries. NumberDict instances support addition,
- and subtraction with other NumberDict instances, and multiplication
- and division by scalars.
- """
-
- def __getitem__(self, item):
- try:
- return dict.__getitem__(self, item)
- except KeyError:
- return 0
-
- def __coerce__(self, other):
- if type(other) == type({}):
- other = NumberDict(other)
- return self, other
-
- def __add__(self, other):
- sum_dict = NumberDict()
- for key in self.keys():
- sum_dict[key] = self[key]
- for key in other.keys():
- sum_dict[key] = sum_dict[key] + other[key]
- return sum_dict
-
- def __sub__(self, other):
- sum_dict = NumberDict()
- for key in self.keys():
- sum_dict[key] = self[key]
- for key in other.keys():
- sum_dict[key] = sum_dict[key] - other[key]
- return sum_dict
-
- def __mul__(self, other):
- new = NumberDict()
- for key in self.keys():
- new[key] = other*self[key]
- return new
- __rmul__ = __mul__
-
- def __div__(self, other):
- new = NumberDict()
- for key in self.keys():
- new[key] = self[key]/other
- return new
-
-import numpy.oldnumeric
-def int_sum(a, axis=0):
- return numpy.oldnumeric.add.reduce(a, axis)
-def zeros_st(shape, other):
- return numpy.oldnumeric.zeros(shape, dtype=other.dtype)
-from numpy import ndarray as array_type
-
-
-import re, string
-
-
-# Class definitions
-
-class PhysicalQuantity:
-
- """
- Physical quantity with units
-
- PhysicalQuantity instances allow addition, subtraction,
- multiplication, and division with each other as well as
- multiplication, division, and exponentiation with numbers.
- Addition and subtraction check that the units of the two operands
- are compatible and return the result in the units of the first
- operand. A limited set of mathematical functions (from module
- Numeric) is applicable as well:
-
- - sqrt: equivalent to exponentiation with 0.5.
-
- - sin, cos, tan: applicable only to objects whose unit is
- compatible with 'rad'.
-
- See the documentation of the PhysicalQuantities module for a list
- of the available units.
-
- Here is an example on usage:
-
- >>> from PhysicalQuantities import PhysicalQuantity as p # short hand
- >>> distance1 = p('10 m')
- >>> distance2 = p('10 km')
- >>> total = distance1 + distance2
- >>> total
- PhysicalQuantity(10010.0,'m')
- >>> total.convertToUnit('km')
- >>> total.getValue()
- 10.01
- >>> total.getUnitName()
- 'km'
- >>> total = total.inBaseUnits()
- >>> total
- PhysicalQuantity(10010.0,'m')
- >>>
- >>> t = p(314159., 's')
- >>> # convert to days, hours, minutes, and second:
- >>> t2 = t.inUnitsOf('d','h','min','s')
- >>> t2_print = ' '.join([str(i) for i in t2])
- >>> t2_print
- '3.0 d 15.0 h 15.0 min 59.0 s'
- >>>
- >>> e = p('2.7 Hartree*Nav')
- >>> e.convertToUnit('kcal/mol')
- >>> e
- PhysicalQuantity(1694.2757596034764,'kcal/mol')
- >>> e = e.inBaseUnits()
- >>> str(e)
- '7088849.77818 kg*m**2/s**2/mol'
- >>>
- >>> freeze = p('0 degC')
- >>> freeze = freeze.inUnitsOf ('degF')
- >>> str(freeze)
- '32.0 degF'
- >>>
- """
-
- def __init__(self, *args):
- """
- There are two constructor calling patterns:
-
- 1. PhysicalQuantity(value, unit), where value is any number
- and unit is a string defining the unit
-
- 2. PhysicalQuantity(value_with_unit), where value_with_unit
- is a string that contains both the value and the unit,
- i.e. '1.5 m/s'. This form is provided for more convenient
- interactive use.
-
- @param args: either (value, unit) or (value_with_unit,)
- @type args: (number, C{str}) or (C{str},)
- """
- if len(args) == 2:
- self.value = args[0]
- self.unit = _findUnit(args[1])
- else:
- s = string.strip(args[0])
- match = PhysicalQuantity._number.match(s)
- if match is None:
- raise TypeError('No number found')
- self.value = string.atof(match.group(0))
- self.unit = _findUnit(s[len(match.group(0)):])
-
- _number = re.compile('[+-]?[0-9]+(\\.[0-9]*)?([eE][+-]?[0-9]+)?')
-
- def __str__(self):
- return str(self.value) + ' ' + self.unit.name()
-
- def __repr__(self):
- return (self.__class__.__name__ + '(' + `self.value` + ',' +
- `self.unit.name()` + ')')
-
- def _sum(self, other, sign1, sign2):
- if not isPhysicalQuantity(other):
- raise TypeError('Incompatible types')
- new_value = sign1*self.value + \
- sign2*other.value*other.unit.conversionFactorTo(self.unit)
- return self.__class__(new_value, self.unit)
-
- def __add__(self, other):
- return self._sum(other, 1, 1)
-
- __radd__ = __add__
-
- def __sub__(self, other):
- return self._sum(other, 1, -1)
-
- def __rsub__(self, other):
- return self._sum(other, -1, 1)
-
- def __cmp__(self, other):
- diff = self._sum(other, 1, -1)
- return cmp(diff.value, 0)
-
- def __mul__(self, other):
- if not isPhysicalQuantity(other):
- return self.__class__(self.value*other, self.unit)
- value = self.value*other.value
- unit = self.unit*other.unit
- if unit.isDimensionless():
- return value*unit.factor
- else:
- return self.__class__(value, unit)
-
- __rmul__ = __mul__
-
- def __div__(self, other):
- if not isPhysicalQuantity(other):
- return self.__class__(self.value/other, self.unit)
- value = self.value/other.value
- unit = self.unit/other.unit
- if unit.isDimensionless():
- return value*unit.factor
- else:
- return self.__class__(value, unit)
-
- def __rdiv__(self, other):
- if not isPhysicalQuantity(other):
- return self.__class__(other/self.value, pow(self.unit, -1))
- value = other.value/self.value
- unit = other.unit/self.unit
- if unit.isDimensionless():
- return value*unit.factor
- else:
- return self.__class__(value, unit)
-
- def __pow__(self, other):
- if isPhysicalQuantity(other):
- raise TypeError('Exponents must be dimensionless')
- return self.__class__(pow(self.value, other), pow(self.unit, other))
-
- def __rpow__(self, other):
- raise TypeError('Exponents must be dimensionless')
-
- def __abs__(self):
- return self.__class__(abs(self.value), self.unit)
-
- def __pos__(self):
- return self
-
- def __neg__(self):
- return self.__class__(-self.value, self.unit)
-
- def __nonzero__(self):
- return self.value != 0
-
- def convertToUnit(self, unit):
- """
- Change the unit and adjust the value such that
- the combination is equivalent to the original one. The new unit
- must be compatible with the previous unit of the object.
-
- @param unit: a unit
- @type unit: C{str}
- @raise TypeError: if the unit string is not a know unit or a
- unit incompatible with the current one
- """
- unit = _findUnit(unit)
- self.value = _convertValue (self.value, self.unit, unit)
- self.unit = unit
-
- def inUnitsOf(self, *units):
- """
- Express the quantity in different units. If one unit is
- specified, a new PhysicalQuantity object is returned that
- expresses the quantity in that unit. If several units
- are specified, the return value is a tuple of
- PhysicalObject instances with with one element per unit such
- that the sum of all quantities in the tuple equals the the
- original quantity and all the values except for the last one
- are integers. This is used to convert to irregular unit
- systems like hour/minute/second.
-
- @param units: one or several units
- @type units: C{str} or sequence of C{str}
- @returns: one or more physical quantities
- @rtype: L{PhysicalQuantity} or C{tuple} of L{PhysicalQuantity}
- @raises TypeError: if any of the specified units are not compatible
- with the original unit
- """
- units = map(_findUnit, units)
- if len(units) == 1:
- unit = units[0]
- value = _convertValue (self.value, self.unit, unit)
- return self.__class__(value, unit)
- else:
- units.sort()
- result = []
- value = self.value
- unit = self.unit
- for i in range(len(units)-1,-1,-1):
- value = value*unit.conversionFactorTo(units[i])
- if i == 0:
- rounded = value
- else:
- rounded = _round(value)
- result.append(self.__class__(rounded, units[i]))
- value = value - rounded
- unit = units[i]
- return tuple(result)
-
- # Contributed by Berthold Hoellmann
- def inBaseUnits(self):
- """
- @returns: the same quantity converted to base units,
- i.e. SI units in most cases
- @rtype: L{PhysicalQuantity}
- """
- new_value = self.value * self.unit.factor
- num = ''
- denom = ''
- for i in xrange(9):
- unit = _base_names[i]
- power = self.unit.powers[i]
- if power < 0:
- denom = denom + '/' + unit
- if power < -1:
- denom = denom + '**' + str(-power)
- elif power > 0:
- num = num + '*' + unit
- if power > 1:
- num = num + '**' + str(power)
- if len(num) == 0:
- num = '1'
- else:
- num = num[1:]
- return self.__class__(new_value, num + denom)
-
- def isCompatible (self, unit):
- """
- @param unit: a unit
- @type unit: C{str}
- @returns: C{True} if the specified unit is compatible with the
- one of the quantity
- @rtype: C{bool}
- """
- unit = _findUnit (unit)
- return self.unit.isCompatible (unit)
-
- def getValue(self):
- """Return value (float) of physical quantity (no unit)."""
- return self.value
-
- def getUnitName(self):
- """Return unit (string) of physical quantity."""
- return self.unit.name()
-
- def sqrt(self):
- return pow(self, 0.5)
-
- def sin(self):
- if self.unit.isAngle():
- return numpy.oldnumeric.sin(self.value * \
- self.unit.conversionFactorTo(_unit_table['rad']))
- else:
- raise TypeError('Argument of sin must be an angle')
-
- def cos(self):
- if self.unit.isAngle():
- return numpy.oldnumeric.cos(self.value * \
- self.unit.conversionFactorTo(_unit_table['rad']))
- else:
- raise TypeError('Argument of cos must be an angle')
-
- def tan(self):
- if self.unit.isAngle():
- return numpy.oldnumeric.tan(self.value * \
- self.unit.conversionFactorTo(_unit_table['rad']))
- else:
- raise TypeError('Argument of tan must be an angle')
-
-
-class PhysicalUnit:
-
- """
- Physical unit
-
- A physical unit is defined by a name (possibly composite), a scaling
- factor, and the exponentials of each of the SI base units that enter into
- it. Units can be multiplied, divided, and raised to integer powers.
- """
-
- def __init__(self, names, factor, powers, offset=0):
- """
- @param names: a dictionary mapping each name component to its
- associated integer power (e.g. C{{'m': 1, 's': -1}})
- for M{m/s}). As a shorthand, a string may be passed
- which is assigned an implicit power 1.
- @type names: C{dict} or C{str}
- @param factor: a scaling factor
- @type factor: C{float}
- @param powers: the integer powers for each of the nine base units
- @type powers: C{list} of C{int}
- @param offset: an additive offset to the base unit (used only for
- temperatures)
- @type offset: C{float}
- """
- if type(names) == type(''):
- self.names = NumberDict()
- self.names[names] = 1
- else:
- self.names = names
- self.factor = factor
- self.offset = offset
- self.powers = powers
-
- def __repr__(self):
- return '<PhysicalUnit ' + self.name() + '>'
-
- __str__ = __repr__
-
- def __cmp__(self, other):
- if self.powers != other.powers:
- raise TypeError('Incompatible units')
- return cmp(self.factor, other.factor)
-
- def __mul__(self, other):
- if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
- raise TypeError("cannot multiply units with non-zero offset")
- if isPhysicalUnit(other):
- return PhysicalUnit(self.names+other.names,
- self.factor*other.factor,
- map(lambda a,b: a+b,
- self.powers, other.powers))
- else:
- return PhysicalUnit(self.names+{str(other): 1},
- self.factor*other,
- self.powers,
- self.offset * other)
-
- __rmul__ = __mul__
-
- def __div__(self, other):
- if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
- raise TypeError("cannot divide units with non-zero offset")
- if isPhysicalUnit(other):
- return PhysicalUnit(self.names-other.names,
- self.factor/other.factor,
- map(lambda a,b: a-b,
- self.powers, other.powers))
- else:
- return PhysicalUnit(self.names+{str(other): -1},
- self.factor/other, self.powers)
-
- def __rdiv__(self, other):
- if self.offset != 0 or (isPhysicalUnit (other) and other.offset != 0):
- raise TypeError("cannot divide units with non-zero offset")
- if isPhysicalUnit(other):
- return PhysicalUnit(other.names-self.names,
- other.factor/self.factor,
- map(lambda a,b: a-b,
- other.powers, self.powers))
- else:
- return PhysicalUnit({str(other): 1}-self.names,
- other/self.factor,
- map(lambda x: -x, self.powers))
-
- def __pow__(self, other):
- if self.offset != 0:
- raise TypeError("cannot exponentiate units with non-zero offset")
- if isinstance(other, int):
- return PhysicalUnit(other*self.names, pow(self.factor, other),
- map(lambda x,p=other: x*p, self.powers))
- if isinstance(other, float):
- inv_exp = 1./other
- rounded = int(numpy.oldnumeric.floor(inv_exp+0.5))
- if abs(inv_exp-rounded) < 1.e-10:
- if reduce(lambda a, b: a and b,
- map(lambda x, e=rounded: x%e == 0, self.powers)):
- f = pow(self.factor, other)
- p = map(lambda x,p=rounded: x/p, self.powers)
- if reduce(lambda a, b: a and b,
- map(lambda x, e=rounded: x%e == 0,
- self.names.values())):
- names = self.names/rounded
- else:
- names = NumberDict()
- if f != 1.:
- names[str(f)] = 1
- for i in range(len(p)):
- names[_base_names[i]] = p[i]
- return PhysicalUnit(names, f, p)
- else:
- raise TypeError('Illegal exponent')
- raise TypeError('Only integer and inverse integer exponents allowed')
-
- def conversionFactorTo(self, other):
- """
- @param other: another unit
- @type other: L{PhysicalUnit}
- @returns: the conversion factor from this unit to another unit
- @rtype: C{float}
- @raises TypeError: if the units are not compatible
- """
- if self.powers != other.powers:
- raise TypeError('Incompatible units')
- if self.offset != other.offset and self.factor != other.factor:
- raise TypeError(('Unit conversion (%s to %s) cannot be expressed ' +
- 'as a simple multiplicative factor') % \
- (self.name(), other.name()))
- return self.factor/other.factor
-
- def conversionTupleTo(self, other): # added 1998/09/29 GPW
- """
- @param other: another unit
- @type other: L{PhysicalUnit}
- @returns: the conversion factor and offset from this unit to
- another unit
- @rtype: (C{float}, C{float})
- @raises TypeError: if the units are not compatible
- """
- if self.powers != other.powers:
- raise TypeError('Incompatible units')
-
- # let (s1,d1) be the conversion tuple from 'self' to base units
- # (ie. (x+d1)*s1 converts a value x from 'self' to base units,
- # and (x/s1)-d1 converts x from base to 'self' units)
- # and (s2,d2) be the conversion tuple from 'other' to base units
- # then we want to compute the conversion tuple (S,D) from
- # 'self' to 'other' such that (x+D)*S converts x from 'self'
- # units to 'other' units
- # the formula to convert x from 'self' to 'other' units via the
- # base units is (by definition of the conversion tuples):
- # ( ((x+d1)*s1) / s2 ) - d2
- # = ( (x+d1) * s1/s2) - d2
- # = ( (x+d1) * s1/s2 ) - (d2*s2/s1) * s1/s2
- # = ( (x+d1) - (d1*s2/s1) ) * s1/s2
- # = (x + d1 - d2*s2/s1) * s1/s2
- # thus, D = d1 - d2*s2/s1 and S = s1/s2
- factor = self.factor / other.factor
- offset = self.offset - (other.offset * other.factor / self.factor)
- return (factor, offset)
-
- def isCompatible (self, other): # added 1998/10/01 GPW
- """
- @param other: another unit
- @type other: L{PhysicalUnit}
- @returns: C{True} if the units are compatible, i.e. if the powers of
- the base units are the same
- @rtype: C{bool}
- """
- return self.powers == other.powers
-
- def isDimensionless(self):
- return not reduce(lambda a,b: a or b, self.powers)
-
- def isAngle(self):
- return self.powers[7] == 1 and \
- reduce(lambda a,b: a + b, self.powers) == 1
-
- def setName(self, name):
- self.names = NumberDict()
- self.names[name] = 1
-
- def name(self):
- num = ''
- denom = ''
- for unit in self.names.keys():
- power = self.names[unit]
- if power < 0:
- denom = denom + '/' + unit
- if power < -1:
- denom = denom + '**' + str(-power)
- elif power > 0:
- num = num + '*' + unit
- if power > 1:
- num = num + '**' + str(power)
- if len(num) == 0:
- num = '1'
- else:
- num = num[1:]
- return num + denom
-
-
-# Type checks
-
-def isPhysicalUnit(x):
- """
- @param x: an object
- @type x: any
- @returns: C{True} if x is a L{PhysicalUnit}
- @rtype: C{bool}
- """
- return hasattr(x, 'factor') and hasattr(x, 'powers')
-
-def isPhysicalQuantity(x):
- """
- @param x: an object
- @type x: any
- @returns: C{True} if x is a L{PhysicalQuantity}
- @rtype: C{bool}
- """
- return hasattr(x, 'value') and hasattr(x, 'unit')
-
-
-# Helper functions
-
-def _findUnit(unit):
- from types import StringTypes
- if isinstance(unit, StringTypes):
- name = string.strip(unit)
- unit = eval(name, _unit_table)
- for cruft in ['__builtins__', '__args__']:
- try: del _unit_table[cruft]
- except: pass
-
- if not isPhysicalUnit(unit):
- raise TypeError(str(unit) + ' is not a unit')
- return unit
-
-def _round(x):
- if numpy.oldnumeric.greater(x, 0.):
- return numpy.oldnumeric.floor(x)
- else:
- return numpy.oldnumeric.ceil(x)
-
-
-def _convertValue (value, src_unit, target_unit):
- (factor, offset) = src_unit.conversionTupleTo(target_unit)
- return (value + offset) * factor
-
-
-# SI unit definitions
-
-_base_names = ['m', 'kg', 's', 'A', 'K', 'mol', 'cd', 'rad', 'sr','EUR']
-
-_base_units = [('m', PhysicalUnit('m', 1., [1,0,0,0,0,0,0,0,0,0])),
- ('g', PhysicalUnit('g', 0.001, [0,1,0,0,0,0,0,0,0,0])),
- ('s', PhysicalUnit('s', 1., [0,0,1,0,0,0,0,0,0,0])),
- ('A', PhysicalUnit('A', 1., [0,0,0,1,0,0,0,0,0,0])),
- ('K', PhysicalUnit('K', 1., [0,0,0,0,1,0,0,0,0,0])),
- ('mol', PhysicalUnit('mol', 1., [0,0,0,0,0,1,0,0,0,0])),
- ('cd', PhysicalUnit('cd', 1., [0,0,0,0,0,0,1,0,0,0])),
- ('rad', PhysicalUnit('rad', 1., [0,0,0,0,0,0,0,1,0,0])),
- ('sr', PhysicalUnit('sr', 1., [0,0,0,0,0,0,0,0,1,0])),
- ('EUR', PhysicalUnit('EUR', 1., [0,0,0,0,0,0,0,0,0,1])),
- ]
-
-_prefixes = [('Y', 1.e24),
- ('Z', 1.e21),
- ('E', 1.e18),
- ('P', 1.e15),
- ('T', 1.e12),
- ('G', 1.e9),
- ('M', 1.e6),
- ('k', 1.e3),
- ('h', 1.e2),
- ('da', 1.e1),
- ('d', 1.e-1),
- ('c', 1.e-2),
- ('m', 1.e-3),
- ('mu', 1.e-6),
- ('n', 1.e-9),
- ('p', 1.e-12),
- ('f', 1.e-15),
- ('a', 1.e-18),
- ('z', 1.e-21),
- ('y', 1.e-24),
- ]
-
-_unit_table = {}
-
-for unit in _base_units:
- _unit_table[unit[0]] = unit[1]
-
-_help = []
-
-def _addUnit(name, unit, comment=''):
- if _unit_table.has_key(name):
- raise KeyError, 'Unit ' + name + ' already defined'
- if comment:
- _help.append((name, comment, unit))
- if type(unit) == type(''):
- unit = eval(unit, _unit_table)
- for cruft in ['__builtins__', '__args__']:
- try: del _unit_table[cruft]
- except: pass
- unit.setName(name)
- _unit_table[name] = unit
-
-def _addPrefixed(unit):
- _help.append('Prefixed units for %s:' % unit)
- _prefixed_names = []
- for prefix in _prefixes:
- name = prefix[0] + unit
- _addUnit(name, prefix[1]*_unit_table[unit])
- _prefixed_names.append(name)
- _help.append(', '.join(_prefixed_names))
-
-
-# SI derived units; these automatically get prefixes
-_help.append('SI derived units; these automatically get prefixes:\n' + \
- ', '.join([prefix + ' (%.0E)' % value for prefix, value in _prefixes]) + \
- '\n')
-
-
-_unit_table['kg'] = PhysicalUnit('kg', 1., [0,1,0,0,0,0,0,0,0,0])
-
-_addUnit('Hz', '1/s', 'Hertz')
-_addUnit('N', 'm*kg/s**2', 'Newton')
-_addUnit('Pa', 'N/m**2', 'Pascal')
-_addUnit('J', 'N*m', 'Joule')
-_addUnit('W', 'J/s', 'Watt')
-_addUnit('C', 's*A', 'Coulomb')
-_addUnit('V', 'W/A', 'Volt')
-_addUnit('F', 'C/V', 'Farad')
-_addUnit('ohm', 'V/A', 'Ohm')
-_addUnit('S', 'A/V', 'Siemens')
-_addUnit('Wb', 'V*s', 'Weber')
-_addUnit('T', 'Wb/m**2', 'Tesla')
-_addUnit('H', 'Wb/A', 'Henry')
-_addUnit('lm', 'cd*sr', 'Lumen')
-_addUnit('lx', 'lm/m**2', 'Lux')
-_addUnit('Bq', '1/s', 'Becquerel')
-_addUnit('Gy', 'J/kg', 'Gray')
-_addUnit('Sv', 'J/kg', 'Sievert')
-
-del _unit_table['kg']
-
-for unit in _unit_table.keys():
- _addPrefixed(unit)
-
-# Fundamental constants
-_help.append('Fundamental constants:')
-
-_unit_table['pi'] = numpy.oldnumeric.pi
-_addUnit('c', '299792458.*m/s', 'speed of light')
-_addUnit('mu0', '4.e-7*pi*N/A**2', 'permeability of vacuum')
-_addUnit('eps0', '1/mu0/c**2', 'permittivity of vacuum')
-_addUnit('Grav', '6.67259e-11*m**3/kg/s**2', 'gravitational constant')
-_addUnit('hplanck', '6.6260755e-34*J*s', 'Planck constant')
-_addUnit('hbar', 'hplanck/(2*pi)', 'Planck constant / 2pi')
-_addUnit('e', '1.60217733e-19*C', 'elementary charge')
-_addUnit('me', '9.1093897e-31*kg', 'electron mass')
-_addUnit('mp', '1.6726231e-27*kg', 'proton mass')
-_addUnit('Nav', '6.0221367e23/mol', 'Avogadro number')
-_addUnit('k', '1.380658e-23*J/K', 'Boltzmann constant')
-
-# Time units
-_help.append('Time units:')
-
-_addUnit('min', '60*s', 'minute')
-_addUnit('h', '60*min', 'hour')
-_addUnit('d', '24*h', 'day')
-_addUnit('wk', '7*d', 'week')
-_addUnit('yr', '365.25*d', 'year')
-
-# Length units
-_help.append('Length units:')
-
-_addUnit('inch', '2.54*cm', 'inch')
-_addUnit('ft', '12*inch', 'foot')
-_addUnit('yd', '3*ft', 'yard')
-_addUnit('mi', '5280.*ft', '(British) mile')
-_addUnit('nmi', '1852.*m', 'Nautical mile')
-_addUnit('Ang', '1.e-10*m', 'Angstrom')
-_addUnit('lyr', 'c*yr', 'light year')
-_addUnit('AU'...
[truncated message content] |
|
From: <zk...@us...> - 2010-02-12 16:22:13
|
Revision: 667
http://pyphant.svn.sourceforge.net/pyphant/?rev=667&view=rev
Author: zklaus
Date: 2010-02-12 16:21:58 +0000 (Fri, 12 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Path issues on Windows machines
Fix: KMVisualizer
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/Helpers.py
trunk/src/pyphant/pyphant/core/KnowledgeManager.py
trunk/src/pyphant/pyphant/core/KnowledgeNode.py
trunk/src/pyphant/pyphant/core/WebInterface.py
trunk/src/pyphant/pyphant/visualizers/KMVisualizer.py
trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
Modified: trunk/src/pyphant/pyphant/core/Helpers.py
===================================================================
--- trunk/src/pyphant/pyphant/core/Helpers.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/core/Helpers.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -29,7 +29,7 @@
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-def getPyphantPath(subdir = '/'):
+def getPyphantPath(subdir = ''):
"""
returns full pyphant path with optional subdirectory
subdir -- subdirectory that is created if it does not exist already,
@@ -37,20 +37,11 @@
"""
import os
homedir = os.path.expanduser('~')
- if not subdir.startswith('/'):
- subdir = '/' + subdir
- if not subdir.endswith('/'):
- subdir = subdir + '/'
if homedir == '~':
homedir = os.getcwdu()
- plist = ('/.pyphant' + subdir).split('/')
- makedir = homedir
- path = homedir + '/.pyphant' + subdir
- for p in plist:
- if p != '':
- makedir += "/%s" % (p, )
- if not os.path.isdir(makedir):
- os.mkdir(makedir)
+ path = os.path.join(homedir, '.pyphant', subdir)
+ if not os.path.isdir(path):
+ os.makedirs(path)
return path
def getUsername():
Modified: trunk/src/pyphant/pyphant/core/KnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -58,7 +58,7 @@
CACHE_MAX_SIZE = 256 * 1024 * 1024
# Limit for number of stored DCs in cache:
CACHE_MAX_NUMBER = 100
-KM_PATH = '/KMstorage/'
+KM_PATH = 'KMstorage'
REHDF5 = re.compile(r'..*\.h5$|..*\.hdf$|..*\.hdf5$')
REFMF = re.compile(r'..*\.fmf$')
@@ -67,18 +67,17 @@
Returns a unique filename for the given emd5.
"""
emd5list = urlparse(dcId + '.h5')[2][2:].split('/')
- emd5path = ''
- for p in emd5list[:-2]:
- emd5path += (p + '/')
- emd5path += emd5list[-2][:10] + '/' + emd5list[-2][11:]\
- + '.' + emd5list[-1]
+ emd5path = os.path.join(
+ *(emd5list[:-2] + [emd5list[-2][:10],
+ emd5list[-2][11:] + '.' + emd5list[-1]]))
directory = os.path.dirname(emd5path)
filename = os.path.basename(emd5path)
if temporary:
- subdir = 'tmp/by_emd5/'
+ subdir = os.path.join('tmp', 'by_emd5')
else:
- subdir = 'by_emd5/'
- return getPyphantPath(KM_PATH + subdir + directory) + filename
+ subdir = 'by_emd5'
+ return os.path.join(getPyphantPath(
+ os.path.join(KM_PATH, subdir, directory)), filename)
class DCNotFoundError(Exception):
@@ -157,7 +156,8 @@
self._cache = []
self._cache_size = 0
if KM_DBASE == u'default':
- self.dbase = getPyphantPath('/sqlite3/') + "km_meta.sqlite3"
+ self.dbase = os.path.join(getPyphantPath('sqlite3'),
+ "km_meta.sqlite3")
else:
self.dbase = KM_DBASE
self.any_value = AnyValue()
@@ -165,7 +165,7 @@
wrapper.setup_dbase()
self.node = None # for hooking up a KnowledgeNode
self.uuid = uuid1().urn
- tmpdir = getPyphantPath(KM_PATH + 'tmp/')
+ tmpdir = getPyphantPath(os.path.join(KM_PATH, 'tmp'))
if os.path.isdir(tmpdir):
from shutil import rmtree
try:
@@ -222,11 +222,11 @@
parsed = urlparse(url)
tmp_extension = ''
if temporary:
- tmp_extension = 'tmp/'
- filename = KM_PATH + tmp_extension + 'registered/' + parsed[1] + '/'
- filename += os.path.basename(parsed[2])
- directory = os.path.dirname(filename)
- filename = getPyphantPath(directory) + os.path.basename(filename)
+ tmp_extension = 'tmp'
+ directory = os.path.join(KM_PATH, tmp_extension,
+ 'registered', parsed[1])
+ filename = os.path.join(getPyphantPath(directory),
+ os.path.basename(parsed[2]))
if os.path.exists(filename):
i = 0
directory = os.path.dirname(filename)
@@ -239,7 +239,8 @@
from sys import maxint
while i < maxint:
fill = str(i).zfill(10)
- tryfn = "%s/%s%s.%s" % (directory, fnwoext, fill, ext)
+ tryfn = os.path.join(directory,
+ "%s%s.%s" % (fnwoext, fill, ext))
if os.path.exists(tryfn):
i += 1
else:
Modified: trunk/src/pyphant/pyphant/core/KnowledgeNode.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeNode.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/core/KnowledgeNode.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -205,13 +205,14 @@
self.km = local_km
self.remotes = []
if dbase == u'default':
- self._dbase = getPyphantPath('/sqlite3/') + 'kn_remotes.sqlite3'
+ self._dbase = os.path.join(getPyphantPath('sqlite3'),
+ 'kn_remotes.sqlite3')
else:
self._dbase = dbase
self._restore_remotes()
self._setup_routes()
self._tempdir = mkdtemp(prefix = 'HDF5Wrap')
- tpl_path = pyphant_source_path[0] + '/web/templates/'
+ tpl_path = os.path.join(pyphant_source_path[0], 'web', 'templates')
if not tpl_path in pyphant.core.bottle.TEMPLATE_PATH:
pyphant.core.bottle.TEMPLATE_PATH.append(tpl_path)
from pyphant.core.WebInterface import WebInterface
Modified: trunk/src/pyphant/pyphant/core/WebInterface.py
===================================================================
--- trunk/src/pyphant/pyphant/core/WebInterface.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/core/WebInterface.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -49,6 +49,7 @@
from pyphant.core.KnowledgeManager import DCNotFoundError
from types import StringTypes
from pyphant.core.SQLiteWrapper import SQLiteWrapper
+import os
def cond(condition, results):
@@ -344,7 +345,7 @@
self.enabled = enabled
self.kn = knowledge_node
from pyphant import __path__ as ppath
- self.rootdir = ppath[0] + '/web/'
+ self.rootdir = os.path.join(ppath[0], 'web')
self.url_link = HTMLLink(self.kn.url, self.kn.url).getHTML()
self.menu = HTMLTable(
[[HTMLLink('/search?shorten=True', 'Browse Data Containers'),
@@ -394,13 +395,15 @@
def images(self, filename):
if not self.enabled:
return template('disabled')
- send_file(filename, self.rootdir + 'images/', guessmime=False,
+ send_file(filename, os.path.join(self.rootdir, 'images'),
+ guessmime=False,
mimetype=self.kn.mimetypes.guess_type(filename)[0])
def script(self, filename):
if not self.enabled:
return template('disabled')
- send_file(filename, self.rootdir + 'script/', guessmime=False,
+ send_file(filename, os.path.join(self.rootdir, 'script'),
+ guessmime=False,
mimetype='application/javascript')
def remote_action(self):
@@ -439,7 +442,7 @@
def log(self):
if not self.enabled:
return template('disabled')
- with open(getPyphantPath() + 'pyphant.log') as logfile:
+ with open(os.path.join(getPyphantPath(), 'pyphant.log')) as logfile:
loglines = logfile.readlines()
return template('log', loglines=''.join(loglines), url=self.url_link)
Modified: trunk/src/pyphant/pyphant/visualizers/KMVisualizer.py
===================================================================
--- trunk/src/pyphant/pyphant/visualizers/KMVisualizer.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/visualizers/KMVisualizer.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -52,10 +52,10 @@
km = KM.getInstance()
km.registerDataContainer(DataContainer)
dc_id = DataContainer.id
- if km.isServerRunning():
- url = 'http://%s:%d/request_dc_details?dcid=%s' % (km._http_host,
- km._http_port,
- dc_id)
+ if km.node is not None:
+ if km.node.app.serve:
+ url = '%ssummary?id=%s' \
+ % (km.node.url, dc_id)
webbrowser.open_new_tab(url)
else:
print "ID of registered DC is: " + dc_id
Modified: trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
===================================================================
--- trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-02-05 14:37:50 UTC (rev 666)
+++ trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-02-12 16:21:58 UTC (rev 667)
@@ -39,7 +39,7 @@
import os, os.path, pkg_resources
from pyphant.core.Helpers import getPyphantPath
-LOGDIR = getPyphantPath()[:-1]
+LOGDIR = getPyphantPath()
import logging
# logging.basicConfig(level=logging.DEBUG)
#else:
@@ -133,8 +133,8 @@
self.Bind(wx.EVT_SIZE, self.onSize)
self.compositeWorkerStack=[]
wx.MessageBox("Located log directory at %s.\n"
- "Logging will go to %s/pyphant.log." %
- (LOGDIR,LOGDIR),
+ "Logging will go to %s." %
+ (LOGDIR, os.path.join(LOGDIR, 'pyphant.log')),
"Logging info")
def _initSash(self):
@@ -278,7 +278,6 @@
self.Bind(wx.EVT_MENU, self.onUpdatePyphant, id=nId)
return updateMenu
-
def onUpdatePyphant(self, event):
import pyphant.core.UpdateManager
pyphant.core.UpdateManager.updatePackage(self.updateIds[event.Id])
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-05 14:37:56
|
Revision: 666
http://pyphant.svn.sourceforge.net/pyphant/?rev=666&view=rev
Author: zklaus
Date: 2010-02-05 14:37:50 +0000 (Fri, 05 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Adapted viewOSC.py to new naming convention.
Modified Paths:
--------------
trunk/doc/demo/viewOSC.py
Modified: trunk/doc/demo/viewOSC.py
===================================================================
--- trunk/doc/demo/viewOSC.py 2010-02-02 11:21:41 UTC (rev 665)
+++ trunk/doc/demo/viewOSC.py 2010-02-05 14:37:50 UTC (rev 666)
@@ -208,8 +208,8 @@
minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
- xPos = table[u"horizontal_table_position"]
- yPos = table[u"vertical_table_position"]
+ xPos = table[u"x-position"]
+ yPos = table[u"y-position"]
thickness = table[u"thickness"]
index = curvNo2Index(table[u"pixel"], curvNo)
result = "$%s_{%s}$(%s %s,%s %s)=%s %s" % (thickness[index].shortname,curvNo,
@@ -231,8 +231,8 @@
oscMap = worker.plugMapHeights.getResult()
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
- xPos = table[u"horizontal_table_position"]
- yPos = table[u"vertical_table_position"]
+ xPos = table[u"x-position"]
+ yPos = table[u"y-position"]
index = curvNo2Index(table[u"pixel"], curvNo)
visualizer = ImageVisualizer(oscMap, False)
pylab.plot([xPos.data[index]],[yPos.data[index]],'xk',scalex=False,scaley=False)
@@ -281,8 +281,8 @@
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
index = table[u"pixel"]
- xPos = table[u"horizontal_table_position"]
- yPos = table[u"vertical_table_position"]
+ xPos = table[u"x-position"]
+ yPos = table[u"y-position"]
thickness = table[u"thickness"]
cols = [index, xPos, yPos, thickness]
worker = recipe.getWorker("MRA Exp")
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-02 11:21:47
|
Revision: 665
http://pyphant.svn.sourceforge.net/pyphant/?rev=665&view=rev
Author: zklaus
Date: 2010-02-02 11:21:41 +0000 (Tue, 02 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Fix: Added missing dependencies.
Modified Paths:
--------------
trunk/src/pyphant/setup.py
Modified: trunk/src/pyphant/setup.py
===================================================================
--- trunk/src/pyphant/setup.py 2010-02-02 11:21:26 UTC (rev 664)
+++ trunk/src/pyphant/setup.py 2010-02-02 11:21:41 UTC (rev 665)
@@ -20,6 +20,8 @@
url='http://pyphant.sourceforge.net/',
install_requires=['sogl>=0.2.0',
'egenix-mx-base',
+ 'paste',
+ 'simplejson',
## The following are required, but currently not setuptools enabled.
#'ScientificPython>=2.6',
#'matplotlib>=0.90.1',
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-02 11:21:32
|
Revision: 664
http://pyphant.svn.sourceforge.net/pyphant/?rev=664&view=rev
Author: zklaus
Date: 2010-02-02 11:21:26 +0000 (Tue, 02 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Change: Adapted viewOSC.py to new naming convention.
Fix: wxPyphant not closing correctly
Modified Paths:
--------------
trunk/doc/demo/viewOSC.py
trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
Modified: trunk/doc/demo/viewOSC.py
===================================================================
--- trunk/doc/demo/viewOSC.py 2010-02-01 17:45:00 UTC (rev 663)
+++ trunk/doc/demo/viewOSC.py 2010-02-02 11:21:26 UTC (rev 664)
@@ -174,8 +174,8 @@
minimaPos = worker.plugMra.getResult().inUnitsOf(simulation.dimensions[1])
worker = recipe.getWorker("AddColumn")
table = worker.plugCompute.getResult(subscriber=TextSubscriber("Add Column"))
- xPos = table[u"horizontal_table_position"]
- yPos = table[u"vertical_table_position"]
+ xPos = table[u"x-position"]
+ yPos = table[u"y-position"]
thickness = table[u"thickness"]
index = curvNo2Index(table[u"pixel"], curvNo)
result = "$%s_{%s}$(%s %s,%s %s)=%s %s" % (thickness[index].shortname,curvNo,
Modified: trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
===================================================================
--- trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-02-01 17:45:00 UTC (rev 663)
+++ trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py 2010-02-02 11:21:26 UTC (rev 664)
@@ -301,6 +301,7 @@
self._wxPyphantApp._knowledgeNode.stop()
except AttributeError:
pass
+ self._wxPyphantApp._excframe.Destroy()
self.Destroy()
def editCompositeWorker(self, worker):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2010-02-01 17:45:07
|
Revision: 663
http://pyphant.svn.sourceforge.net/pyphant/?rev=663&view=rev
Author: zklaus
Date: 2010-02-01 17:45:00 +0000 (Mon, 01 Feb 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master: (35 commits)
Enh: Exceptions now displayed in wxPyphant
Enh: Saving recipes w/o results
Enh: Added save as dialog to wxPyphant
Enh: Added batch method to Helpers module
Enh: Added sobel filter to NDImage
Enh: Wrote XMLHandler
Enh: Added preliminary attribute search
Cosm: Nicer web interface
Cosm: Nicer web interface
Enh: Improved web interface
Enh: Improved web interface
Enh: Reimplemented simple web search (dirty)
Fix: Adapted code to fit Python 2.5 and 2.6
Enh: Routed paste logger to pyphant logger
Cosm: Nicer web interface
Fix: Fixed syntax to fit python 2.5
Enh: Added log to web interface
Fix: Bugfix for KN
Fix: Bugfix for KN
Enh: improved KN usage
...
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/H5FileHandler.py
trunk/src/pyphant/pyphant/core/Helpers.py
trunk/src/pyphant/pyphant/core/KnowledgeManager.py
trunk/src/pyphant/pyphant/core/PyTablesPersister.py
trunk/src/pyphant/pyphant/core/SQLiteWrapper.py
trunk/src/pyphant/pyphant/core/WebInterface.py
trunk/src/pyphant/pyphant/tests/TestH5FileHandler.py
trunk/src/pyphant/pyphant/tests/TestSQLiteWrapper.py
trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
trunk/src/workers/ImageProcessing/ImageProcessing/NDImageWorker.py
Added Paths:
-----------
trunk/src/pyphant/pyphant/core/KnowledgeNode.py
trunk/src/pyphant/pyphant/core/RoutingHTTPServer.py
trunk/src/pyphant/pyphant/core/XMLHandler.py
trunk/src/pyphant/pyphant/core/bottle.py
trunk/src/pyphant/pyphant/tests/TestKnowledgeNode.py
trunk/src/pyphant/pyphant/web/
trunk/src/pyphant/pyphant/web/images/
trunk/src/pyphant/pyphant/web/images/disabled.gif
trunk/src/pyphant/pyphant/web/images/favicon.ico
trunk/src/pyphant/pyphant/web/images/offline.gif
trunk/src/pyphant/pyphant/web/images/online.gif
trunk/src/pyphant/pyphant/web/images/pyphant.png
trunk/src/pyphant/pyphant/web/script/
trunk/src/pyphant/pyphant/web/script/LaTeXMathML.js
trunk/src/pyphant/pyphant/web/templates/
trunk/src/pyphant/pyphant/web/templates/back.tpl
trunk/src/pyphant/pyphant/web/templates/disabled.tpl
trunk/src/pyphant/pyphant/web/templates/fieldcontainer.tpl
trunk/src/pyphant/pyphant/web/templates/frontpage.tpl
trunk/src/pyphant/pyphant/web/templates/htmlhead.tpl
trunk/src/pyphant/pyphant/web/templates/log.tpl
trunk/src/pyphant/pyphant/web/templates/mathhead.tpl
trunk/src/pyphant/pyphant/web/templates/message.tpl
trunk/src/pyphant/pyphant/web/templates/ordermathhead.tpl
trunk/src/pyphant/pyphant/web/templates/remotes.tpl
trunk/src/pyphant/pyphant/web/templates/samplecontainer.tpl
trunk/src/pyphant/pyphant/web/templates/search.tpl
Modified: trunk/src/pyphant/pyphant/core/H5FileHandler.py
===================================================================
--- trunk/src/pyphant/pyphant/core/H5FileHandler.py 2010-01-11 21:39:34 UTC (rev 662)
+++ trunk/src/pyphant/pyphant/core/H5FileHandler.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -44,9 +44,20 @@
import logging
import os
from pyphant.core import PyTablesPersister
+from pyphant.core.DataContainer import IndexMarker
+from pyphant.core.Helpers import (utf82uc, emd52dict)
_logger = logging.getLogger("pyphant")
+im = IndexMarker()
+im_id = u"emd5://pyphant/pyphant/0001-01-01_00:00:00.000000/%s.field" \
+ % utf82uc(im.hash)
+im_summary = {'id':im_id, 'longname':utf82uc(im.longname),
+ 'shortname':utf82uc(im.shortname), 'hash':utf82uc(im.hash),
+ 'creator':u'pyphant', 'machine':u'pyphant',
+ 'date':u'0001-01-01_00:00:00.000000',
+ 'unit':1, 'dimensions':[im_id], 'attributes':{}}
+
class H5FileHandler(object):
"""
This class is used to handle IO operations on HDF5 files.
@@ -157,12 +168,13 @@
currDcId = group._v_attrs.TITLE
if len(currDcId) > 0:
tmp = self.loadSummary(currDcId)
- if tmp != u'IndexMarker':
+ if tmp == 'IndexMarker':
+ summary[im_id] = im_summary
+ else:
summary[currDcId] = tmp
elif self.isIndexMarker(dcId):
return u'IndexMarker'
else:
- from pyphant.core.Helpers import (utf82uc, emd52dict)
summary = {}
summary['id'] = dcId
resNode, uriType = self.getNodeAndTypeFromId(dcId)
@@ -187,7 +199,7 @@
dimTable = resNode.dimensions
def filterIndexMarker(emd5):
if self.isIndexMarker(emd5):
- return u'IndexMarker'
+ return im_id
else:
return emd5
dimensions = [filterIndexMarker(row['id']) \
Modified: trunk/src/pyphant/pyphant/core/Helpers.py
===================================================================
--- trunk/src/pyphant/pyphant/core/Helpers.py 2010-01-11 21:39:34 UTC (rev 662)
+++ trunk/src/pyphant/pyphant/core/Helpers.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -131,3 +131,47 @@
retdict['hash'] = emd5_split[5].split('.')[0]
retdict['type'] = emd5_split[5].split('.')[1]
return retdict
+
+def batch(recipe, input, plug, longname, dobatch=True, temporary=False):
+ """
+ Runs the same recipe multiple times for different input data.
+ The return value is either a SampleContainer similar to input
+ with 'emd5' column replaced by results or the resulting
+ DataContainer from plug, if dobatch is set to False.
+ recipe -- CompositeWorker instance
+ input -- SampleContainer with 'emd5' column or any DataContainer if
+ dobatch is set to False
+ plug -- plug contained in recipe to get output from
+ (there has to be exactly one open socket in recipe
+ ascending from plug)
+ longname -- longname of resulting SampleContainer, works only for
+ dobatch == True
+ dobatch -- if set to False, input is treated as a single data source
+ temporary -- whether to register results temporarily, only applies when
+ dobatch is set to True
+ """
+ socket = recipe.getOpenSocketsForPlug(plug)[0]
+ from tools import Emd5Src
+ DummyWorker = Emd5Src.Emd5Src()
+ socket.insert(DummyWorker.getPlugs()[0])
+ DummyWorker.paramSelectby.value = u"enter emd5"
+ from pyphant.core.KnowledgeManager import KnowledgeManager
+ km = KnowledgeManager.getInstance()
+ if dobatch:
+ import copy
+ output = copy.deepcopy(input)
+ index = 0
+ for emd5 in input['emd5'].data:
+ DummyWorker.paramEnteremd5.value = emd5
+ resultDC = plug.getResult()
+ km.registerDataContainer(resultDC, temporary=temporary)
+ output['emd5'].data[index] = resultDC.id
+ index += 1
+ output.longname = longname
+ output.seal()
+ else:
+ km.registerDataContainer(input)
+ DummyWorker.paramEnteremd5.value = input.id
+ output = plug.getResult()
+ socket.pullPlug()
+ return output
Modified: trunk/src/pyphant/pyphant/core/KnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-01-11 21:39:34 UTC (rev 662)
+++ trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -28,12 +28,12 @@
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
from __future__ import with_statement
"""
-This module provides the KnowledgeManager class as well as some helper methods
-for handling the .pyphant directory and some helper classes for the
-KnowledgeManager class.
+This module provides the KnowledgeManager class as well as some helper
+classes.
"""
__id__ = "$Id$"
@@ -42,32 +42,18 @@
# $Source: $
from pyphant.core.singletonmixin import Singleton
-import urllib
-import cgi
import tempfile
-import sys
-import os, os.path
+import os
import logging
-import threading
-from uuid import uuid1
import re
-import time
-from urlparse import urlparse
-from SimpleHTTPServer import SimpleHTTPRequestHandler
-from pyphant.core.H5FileHandler import H5FileHandler
-from pyphant.core.WebInterface import (HTTPAnswer,
- WebInterface,
- KMHTMLParser,
- ThreadedHTTPServer)
+from pyphant.core.H5FileHandler import (H5FileHandler, im_id)
from fmfile import FMFLoader
from pyphant.core.SQLiteWrapper import (SQLiteWrapper, AnyValue)
+from pyphant.core.Helpers import getPyphantPath
+from uuid import uuid1
+from urlparse import urlparse
+import urllib
-WAITING_SECONDS_HTTP_SERVER_STOP = 5
-HTTP_REQUEST_DC_URL_PATH = "/request_dc_url"
-HTTP_REQUEST_KM_ID_PATH = "/request_km_id"
-HTTP_REQUEST_DC_DETAILS_PATH = "/request_dc_details?dcid="
-HTTP_REQUEST_REGISTER_KM = "/request_register_km"
-HTTP_REQUEST_SEARCH = "/request_search"
# Limit for sum(DC.rawDataBytes) for DC in cache:
CACHE_MAX_SIZE = 256 * 1024 * 1024
# Limit for number of stored DCs in cache:
@@ -76,29 +62,6 @@
REHDF5 = re.compile(r'..*\.h5$|..*\.hdf$|..*\.hdf5$')
REFMF = re.compile(r'..*\.fmf$')
-def getPyphantPath(subdir = '/'):
- """
- returns full pyphant path with optional subdirectory
- subdir -- subdirectory that is created if it does not exist already,
- recursive creation of directories is supported also.
- """
- homedir = os.path.expanduser('~')
- if not subdir.startswith('/'):
- subdir = '/' + subdir
- if not subdir.endswith('/'):
- subdir = subdir + '/'
- if homedir == '~':
- homedir = os.getcwdu()
- plist = ('/.pyphant' + subdir).split('/')
- makedir = homedir
- path = homedir + '/.pyphant' + subdir
- for p in plist:
- if p != '':
- makedir += "/%s" % (p, )
- if not os.path.isdir(makedir):
- os.mkdir(makedir)
- return path
-
def getFilenameFromDcId(dcId, temporary=False):
"""
Returns a unique filename for the given emd5.
@@ -118,33 +81,14 @@
return getPyphantPath(KM_PATH + subdir + directory) + filename
-class KnowledgeManagerException(Exception):
- """
- Exception class that is able to store parent exceptions.
- """
- def __init__(self, message, parent_excep = None, *args, **kwds):
- """
- message -- human readable reason for the exception
- parent_excep -- exception that is the reason for throwing this one.
- """
- super(KnowledgeManagerException, self).__init__(message, *args, **kwds)
- self._message = message
- self._parent_excep = parent_excep
+class DCNotFoundError(Exception):
+ pass
- def __str__(self):
- """
- Returns error message with reason from parent exception.
- """
- return self._message + " (reason: %s)" % (str(self._parent_excep), )
- def getParentException(self):
- """
- Returns the parent exception
- """
- return self._parent_excep
-
-
class CachedDC(object):
+ """
+ Class representing a cached DataContainer
+ """
def __init__(self, dc_ref):
self.id = dc_ref.id
self.ref = dc_ref
@@ -155,6 +99,9 @@
class TestCachedDC(object):
+ """
+ Class representing a cache lookup
+ """
def __init__(self, dc_id):
self.id = dc_id
@@ -162,80 +109,73 @@
return self.id == other.id
+KM_DBASE = u'default' # modify for debug purposes
+
+
class KnowledgeManager(Singleton):
"""
Knowledge Manager for Pyphant
=============================
- The ID of a DataContainer object is given by a emd5 string.
+ The ID of a DataContainer object is given by an emd5 string.
Responsibilities:
-----------------
- - register HDF5 files by their URLs
- - register remote knowledge managers by urls
- - share data containers via HTTP, they are requested by id
- - get references for these data containers (local or remote)
- If an operation fails, a KnowledgeManagerException
- will be raised. These exceptions have a method
- .getParentException()
- in order to get additional information about the reason.
- Usage:
- ------
+ - Manage local storage of DataContainers
+ - Resolve IDs to DC instances
+ - Communicate with a KnowledgeNode in order to share Knowledge
+ among other KM instances via HTTP.
+ - Manage and search meta data for DataContainers
+ Usage examples:
+ ---------------
Get a reference to the KnowledgeManager instance, which is a
singleton:
- import pyphant.core.KnowledgeManager as KM
- km = KM.KnowledgeManager.getInstance()
- Optionally: Start HTTP server for sharing data with others by
- km.startServer(<host>,<port>)
+ from pyphant.core.KnowledgeManager import KnowledgeManager
+ km = KnowledgeManager.getInstance()
Register a local HDF5 file:
+ km.registerURL("/some_directory/data.h5")
+ Register and persist a local HDF5 file:
km.registerURL("file:///tmp/data.h5")
- Register a remote HDF5 file:
- km.registerURL("http://example.com/repository/data.h5")
- Register another KnowledgeManager in order to benefit
- from their knowledge (see arguments of .startServer):
- km.registerKnowledgeManager("http://example.com", 8000, True)
- Request data container by its id:
- dc = km.getDataContainer(id)
- Use the data container!
+ Register and persist a remote FMF file:
+ km.registerURL("http://example.com/repository/data.fmf")
+ Request DataContainer by its id:
+ dc = km.getDataContainer(id) # `dc` is an actual instance now
+ For searching meta data see docstring of KM.search() below.
+ How to share Knowledge:
+ Hook up the KM to a KnowledgeNode, see documentation in the
+ KnowledgeNode module.
+ Known issues:
+ ------------
+ KM is NOT thread-safe yet.
"""
def __init__(self):
"""
- Sets the unique id for the KM instance, sets up the DataBase if
- it has not been initialized yet and clears the tmp dir.
+ Sets up the DataBase if it has not been initialized yet,
+ sets up the cache and clears the tmp dir.
+ Sets a uuid to identify the instance.
"""
super(KnowledgeManager, self).__init__()
- self._logger = logging.getLogger("pyphant")
+ self.logger = logging.getLogger("pyphant")
self._cache = []
self._cache_size = 0
- self.H5FileHandlers = {}
- self._remoteKMs = {} # key:id, value:url
- self._server = None
- self._server_id = uuid1()
- self.web_interface = WebInterface(self, True)
- self.dbase = getPyphantPath('/sqlite3/') + "km_meta.sqlite3"
+ if KM_DBASE == u'default':
+ self.dbase = getPyphantPath('/sqlite3/') + "km_meta.sqlite3"
+ else:
+ self.dbase = KM_DBASE
self.any_value = AnyValue()
with SQLiteWrapper(self.dbase) as wrapper:
wrapper.setup_dbase()
+ self.node = None # for hooking up a KnowledgeNode
+ self.uuid = uuid1().urn
tmpdir = getPyphantPath(KM_PATH + 'tmp/')
if os.path.isdir(tmpdir):
from shutil import rmtree
try:
rmtree(tmpdir)
except OSError:
- print "Could not delete '%s'." % (tmpdir, )
+ self.logger.warn("Could not delete '%s'." % tmpdir)
- def tearDown(self):
- """
- Stops the HTTP server
- """
- if self.isServerRunning():
- self.stopServer()
-
def hasDataContainer(self, dcid):
"""
Returns whether the given DC is stored locally.
- Never use this method in a 'with SQLiteWrapper(...) as wrapper'
- statement! Use wrapper.has_entry(dcid) instead if you already
- have a wrapper at your hands or you may end up in a sqlite3 locking
- loop.
"""
with SQLiteWrapper(self.dbase) as wrapper:
has_entry = wrapper.has_entry(dcid)
@@ -263,135 +203,11 @@
summaryDict = h5fh.loadSummary()
with SQLiteWrapper(self.dbase) as wrapper:
for dcId, summary in summaryDict.items():
- if not wrapper.has_entry(dcId):
+ if dcId == im_id:
+ wrapper.set_entry(summary, None, temporary)
+ else:
wrapper.set_entry(summary, filename, temporary)
- def _getServerURL(self):
- """
- Returns the URL of the HTTP server.
- """
- if self._server is None:
- return None
- return "http://%s:%d" % (self._http_host, self._http_port)
-
- def getServerId(self):
- """
- Returns uniqe id of the KnowledgeManager as uuid URN.
- """
- return self._server_id.urn
-
- def startServer(self, host = '127.0.0.1', port = 8000,
- provide_web_frontend = False):
- """
- Starts the HTTP server. When the server was running already,
- it is restartet with the new parameters.
- A temporary directory is generated in order to
- save temporary HDF5 files.
- The data may be announced to other KnowledgeManagers.
- host -- full qualified domain name or IP address under which
- server can be contacted via HTTP, default: '127.0.0.1'
- port -- port of HTTP server (integer), default: 8000
- provide_web_frontend -- whether to provide web frontend, default: False
- """
- logger = self._logger
- if self.isServerRunning():
- logger.warn("Server is running at host %s, port %d already. "
- "Stopping server...", self._http_host, self._http_port)
- self.stopServer()
- self._http_host = host
- self._http_port = port
- self._http_dir = tempfile.mkdtemp(prefix = 'pyphant-knowledgemanager')
- self._server = ThreadedHTTPServer((host, port), KMHTTPRequestHandler)
- class _HTTPServerThread(threading.Thread):
- def run(other):
- self._server.start()
- self._http_server_thread = _HTTPServerThread()
- self._http_server_thread.start()
- self._logger.debug("Started HTTP server. Host: %s, port: %d, "
- "temp dir: %s", host, port, self._http_dir)
- self.web_interface.disabled = not provide_web_frontend
-
- def stopServer(self):
- """
- Stops the HTTP server. The temporary directory is removed.
- """
- logger = self._logger
- if self.isServerRunning():
- self.web_interface.disabled = True
- self._server.stop_server = True
- # do fake request
- try:
- urllib.urlopen(self._getServerURL())
- except:
- logger.warn("Fake HTTP request failed when stopping HTTP "
- "server.")
- logger.info("Waiting for HTTP server thread to die...")
- self._http_server_thread.join(WAITING_SECONDS_HTTP_SERVER_STOP)
- if self._http_server_thread.isAlive():
- logger.warn("HTTP server thread could not be stopped.")
- else:
- logger.info("HTTP server has been stopped.")
- self._server = None
- self._http_host = None
- self._http_port = None
- try:
- logger.debug("Deleting temporary directory '%s'..",
- self._http_dir)
- os.removedirs(self._http_dir)
- except Exception:
- logger.warn("Failed to delete temporary directory '%s'.",
- self._http_dir)
- self._http_dir = None
- else:
- self._logger.warn("HTTP server should be stopped but isn't "
- "running.")
-
- def isServerRunning(self):
- """
- Returns whether HTTP server is running.
- """
- return self._server is not None
-
- def registerKnowledgeManager(self, host, port = 8000,
- share_knowledge = False):
- """
- Registers a knowledge manager. The remote KnowledgeManager is
- contacted immediately in order to save its unique ID.
- host -- full qualified domain name or IP address at which
- server can be contacted via HTTP
- port -- port of HTTP server (integer), default: 8000
- share_knowledge -- local knowledge is made available to the remote KM
- when set to True and the HTTP server is running at
- the local KM, default: False
- """
- logger = self._logger
- try:
- km_url = "http://%s:%d" % (host, port)
- # get unique id from KM via HTTP
- logger.debug("Requesting ID from Knowledgemanager with URL '%s'...",
- km_url)
- # request url for given id over http
- local_km_host = ''
- local_km_port = ''
- if self.isServerRunning() and share_knowledge:
- local_km_host = self._http_host
- local_km_port = str(self._http_port)
- post_data = urllib.urlencode({'kmhost':local_km_host,
- 'kmport':local_km_port})
- answer = urllib.urlopen(km_url + HTTP_REQUEST_KM_ID_PATH, post_data)
- logger.debug("Info from HTTP answer: %s", answer.info())
- htmltext = answer.read()
- parser = KMHTMLParser()
- parser.feed(htmltext)
- km_id = parser.headitems['pyphant']['kmid'].strip()
- answer.close()
- logger.debug("KM ID read from HTTP answer: %s", km_id)
- except Exception, excep:
- raise KnowledgeManagerException(
- "Couldn't get ID for knowledge manager under URL %s."
- % (km_url, ), excep)
- self._remoteKMs[km_id] = km_url
-
def registerURL(self, url, temporary=False):
"""
Registers an HDF5 or FMF file downloadable from given URL and stores it
@@ -420,7 +236,8 @@
fnwoext = ''
for part in split:
fnwoext += (part + '.')
- while i < sys.maxint:
+ from sys import maxint
+ while i < maxint:
fill = str(i).zfill(10)
tryfn = "%s/%s%s.%s" % (directory, fnwoext, fill, ext)
if os.path.exists(tryfn):
@@ -428,17 +245,17 @@
else:
filename = tryfn
break
- self._logger.info("Retrieving url '%s'..." % (url, ))
- self._logger.info("Using local file '%s'." % (filename, ))
+ self.logger.info("Retrieving url '%s'..." % url)
+ self.logger.info("Using local file '%s'." % filename)
savedto, headers = urllib.urlretrieve(url, filename)
- self._logger.info("Header information: %s", (str(headers), ))
if REFMF.match(filename.lower()) != None:
self.registerFMF(filename, temporary)
elif REHDF5.match(filename.lower()) != None:
self.registerH5(filename, temporary)
else:
- raise KnowledgeManagerException('Filetype unknown: %s'
- % (filename, ))
+ msg = "Could not guess type of '%s'" % url
+ self.logger.error(msg)
+ raise ValueError(msg)
def registerDataContainer(self, dc, temporary=False):
"""
@@ -455,8 +272,9 @@
your hard drive.
"""
if dc.id == None:
- raise KnowledgeManagerException("Invalid id for DataContainer '"\
- + dc.longname + "'")
+ msg = "Missing id for DataContainer. DC has not been sealed."
+ self.logger.error(msg)
+ raise ValueError(msg)
if not self.hasDataContainer(dc.id):
filename = getFilenameFromDcId(dc.id, temporary)
handler = self.getH5FileHandler(filename, 'w')
@@ -476,101 +294,6 @@
self.registerDataContainer(sc, temporary)
return sc.id
- def _getDCURLFromRemoteKMs(self, query_dict):
- """
- Returns URL for a DataContainer by requesting remote
- KnowledgeManagers.
- query_dict -- see _getDataContainerURL
- """
- logger = self._logger
- # add this KM to query
- query_dict['lastkmidindex'] += 1
- query_dict['kmid%d' % query_dict['lastkmidindex']] = self.getServerId()
- # ask every remote KnowledgeManager for id
- logger.debug("Requesting knowledge managers for DC id '%s'..."
- % (query_dict['dcid'], ))
- dc_url = None
- for km_id, km_url in self._remoteKMs.iteritems():
- if not (km_id in query_dict.values()): #<-- TODO: exclude wrong keys
- logger.debug("Requesting Knowledgemanager with ID '%s' and \
-URL '%s'...", km_id, km_url)
- # request url for given id over http
- try:
- data = urllib.urlencode(query_dict)
- logger.debug("URL encoded query: %s", data)
- answer = urllib.urlopen(km_url + HTTP_REQUEST_DC_URL_PATH,
- data)
- code = int(answer.headers.dict['code'])
- if code < 400:
- parser = KMHTMLParser()
- htmltext = answer.read()
- parser.feed(htmltext)
- dc_url = parser.headitems['pyphant']['hdf5url'].strip()
- logger.debug("URL for id read from HTTP answer: %s",
- dc_url)
- break
- elif code == 404:
- # update query_dict:
- parser = KMHTMLParser()
- htmltext = answer.read()
- parser.feed(htmltext)
- query_dict.clear()
- for k in parser.headitems['pyphant']:
- query_dict[k] =\
- parser.headitems['pyphant'][k].strip()
- query_dict['lastkmidindex']\
- = int(query_dict['lastkmidindex'])
- logger.debug("Code 404 from '%s', updated query: %s",
- km_url, str(query_dict))
- else:
- # message for everyone: do not ask this KM again
- query_dict['lastkmidindex'] += 1
- query_dict['kmid%d' % (query_dict['lastkmidindex'], )]\
- = km_id
- except:
- logger.debug("Could not contact KM with ID '%s'", km_id)
- # message for everyone: do not ask this KM again
- query_dict['lastkmidindex'] += 1
- query_dict['kmid%d' % (query_dict['lastkmidindex'], )]\
- = km_id
- finally:
- answer.close()
- return dc_url
-
- def _getDataContainerURL(self, query_dict):
- """
- Returns a URL from which a DataContainer can be downloaded.
- The server must be running before calling this method.
- query_dict -- dict of DC ID to get and KnowledgeManager IDs
- which shouldn't be asked.
- e.g.: {'dcid':'somedcid',
- 'lastkmidindex:1',
- 'kmid0':'someid',
- 'kmid1':'anotherid'}
- query_dict is extended by this method in order to
- exclude KMs recursively.
- """
- assert self.isServerRunning(), "Server is not running."
- dc_id = query_dict['dcid']
- if self.hasDataContainer(dc_id):
- dc = self.getDataContainer(dc_id, True, False)
- # Wrap data container in temporary HDF5 file
- osFileId, filename = tempfile.mkstemp(suffix = '.h5',
- prefix = 'dcrequest-',
- dir = self._http_dir)
- os.close(osFileId)
- handler = H5FileHandler(filename, 'w')
- with handler:
- handler.saveDataContainer(dc)
- dc_url = self._getServerURL() + "/" + os.path.basename(filename)
- else:
- try:
- dc_url = self._getDCURLFromRemoteKMs(query_dict)
- except Exception, excep:
- raise KnowledgeManagerException(
- "URL for DC ID '%s' not found." % (dc_id, ), excep)
- return dc_url
-
def getDCFromCache(self, dc_id, filename):
"""
Returns a DC instance from cache or local storage.
@@ -606,7 +329,7 @@
self._cache.append(cache_item)
self._cache_size += cache_item.size
- def getDataContainer(self, dc_id, use_cache = True, try_remote = True):
+ def getDataContainer(self, dc_id, use_cache=True, try_remote=True):
"""
Returns DataContainer matching the given id.
dc_id -- Unique ID of the DataContainer (emd5)
@@ -626,21 +349,14 @@
with self.getH5FileHandler(filename) as handler:
dc = handler.loadDataContainer(dc_id)
return dc
- elif try_remote:
- dc_url = self._getDCURLFromRemoteKMs({'dcid':dc_id,
- 'lastkmidindex':-1})
- if dc_url == None:
- raise KnowledgeManagerException("DC ID '%s' is unknown."
- % (dc_id,))
- filename = getFilenameFromDcId(dc_id)
- urllib.urlretrieve(dc_url, filename)
- self.registerH5(filename)
- with self.getH5FileHandler(filename) as handler:
- dc = handler.loadDataContainer(dc_id)
- return dc
- else:
- raise KnowledgeManagerException("DC ID '%s' is unknown."
- % (dc_id, ))
+ elif try_remote and self.node != None:
+ try:
+ return self.node.get_datacontainer(dc_id)
+ except DCNotFoundError:
+ pass
+ msg = "Could not find DC with id '%s'." % dc_id
+ self.logger.error(msg)
+ raise DCNotFoundError(msg)
def getEmd5List(self):
"""
@@ -652,7 +368,52 @@
def search(self, result_keys, search_dict={}, order_by=None,
order_asc=True, limit=-1, offset=0, distinct=False):
"""
- See SQLiteWrapper.get_andsearch_result()
+ returns a list of tuples filled with values of the result keys
+ matching the constraints of search_dict.
+ Arguments:
+ - result_keys: List (of length >= 1) of keys to include in the
+ result tuples.
+ - search_dict: Dict mapping keys to constraint values.
+ Use empty dict for no constraints at all
+ possible keys: values (used relational operator[, type constraint]):
+ 'longname': str types (==)
+ 'shortname': str types (==)
+ 'machine': str types (==)
+ 'creator: str types (==)
+ 'date_from:' str types:
+ YYYY[-MM[-DD[_hh:[mm:[ss[.s[s[s[s[s[s]]]]]]]]]]] (>=)
+ 'date_to:' str types:
+ YYYY[-MM[-DD[_hh:[mm:[ss[.s[s[s[s[s[s]]]]]]]]]]] (<)
+ 'hash': str types (==)
+ 'id': str types: emd5 (==)
+ 'type': 'field' or 'sample' (==)
+ 'attributes': dict mapping attr. key to attr. value (==)
+ use (SQLiteWrapper instance).any_value
+ or (KM instance).any_value to skip value check
+ 'storage': str types (==)
+ 'unit': PhysicalUnit or number or PhysicalQuantity (==, FC only)
+ 'dimensions': list of FC search dicts
+ (see above definitions, FC only)
+ 'columns': list of FC search dicts (see above definitions, SC only)
+ - order_by: element of result_keys to order the results by
+ or None for no special ordering
+ - order_asc: whether to order ascending
+ - limit: maximum number of results to return,
+ set to -1 for no limit, default: -1
+ - offset: number of search results to skip, default: 0
+ - distinct: flag that indicates whether the result list
+ should only contain distinct tuples.
+ Usage Examples:
+ Get list of all longnames:
+ get_andsearch_result(['longname'], distinct=True)
+ --> [('name1', ), ('name2', ), ...]
+ Get id and shortname of all FCs that are parametrized by
+ a time dimension along the primary axis:
+ tunit = PhysicalQuantity(1, 's')
+ get_andsearch_result(['id', 'shortname'],
+ {'type':'field',
+ 'dimensions':[{'unit':tunit}]})
+ --> [('emd5_1', 'name_1'), ('emd5_2', 'name_2'), ...]
"""
with SQLiteWrapper(self.dbase) as wrapper:
return wrapper.get_andsearch_result(
@@ -665,7 +426,6 @@
the given DC.
"""
with SQLiteWrapper(self.dbase) as wrapper:
- # TODO: usage of rowwrapper is not optimal in performance
rowwrapper = wrapper[dc_id]
keys = list(SQLiteWrapper.all_keys)
if dc_id.endswith('field'):
@@ -675,184 +435,3 @@
keys.remove('dimensions')
summary = dict([(key, rowwrapper[key]) for key in keys])
return summary
-
-
-class KMHTTPRequestHandler(SimpleHTTPRequestHandler):
- """
- Helper class for KnowledgeManager that handles HTTP requests.
- """
- _km = KnowledgeManager.getInstance()
- _logger = logging.getLogger("pyphant")
- def send_response(self, code, message = None):
- """
- Sends HTTP status code and an optional message via HTTP headers.
- code -- HTTP status code e.g. 404: File not found
- message -- optional reason for the given code
- """
- self.log_request(code)
- if message is None:
- if code in self.responses:
- message = self.responses[code][0]
- else:
- message = ''
- if self.request_version != 'HTTP/0.9':
- self.wfile.write("%s %d %s\r\n" % (self.protocol_version, code,
- message))
- self.send_header('Server', self.version_string())
- self.send_header('Date', self.date_time_string())
- #for older versions of urllib.urlopen which do not support
- #.getcode() method
- self.send_header('code', str(code))
-
- def do_POST(self):
- """
- Handles HTTP POST requests.
- """
- self._logger.debug("POST request from client (host,port): %s",
- self.client_address)
- self._logger.debug("POST request path: %s", self.path)
- if self.path == HTTP_REQUEST_DC_URL_PATH:
- httpanswer = self._do_POST_request_dc_url()
- elif self.path == HTTP_REQUEST_KM_ID_PATH:
- httpanswer = self._do_POST_request_km_id()
- elif self.path == HTTP_REQUEST_REGISTER_KM:
- httpanswer = self._do_POST_request_register_km()
- else:
- code = 400
- message = "Unknown request path '%s'." % (self.path, )
- httpanswer = HTTPAnswer(code, message)
- httpanswer.sendTo(self)
-
- def _do_POST_request_km_id(self):
- """
- Returns the KnowledgeManager ID via HTTP in the HTML head as
- "<pyphant kmid = '...'>".
- """
- km = KMHTTPRequestHandler._km
- if self.headers.has_key('content-length'):
- length = int( self.headers['content-length'] )
- query = self.rfile.read(length)
- query_dict = cgi.parse_qs(query)
- remote_host = ''
- remote_port = ''
- try:
- remote_host = query_dict['kmhost'][0]
- remote_port = query_dict['kmport'][0]
- except:
- #self._logger.info("Remote knowledge is not being shared.")
- pass
- if remote_host != '' and remote_port != '':
- km.registerKnowledgeManager(remote_host, int(remote_port),
- False)
- self._logger.debug("Returning ID '%s'...", km.getServerId())
- return km.web_interface.get_kmid(km.getServerId())
-
- def _do_POST_request_dc_url(self):
- """
- Returns an URL for a given DataContainer ID via HTTP in the HTML head as
- "<pyphant hdf5url = '...'>".
- """
- if self.headers.has_key('content-length'):
- length = int( self.headers['content-length'] )
- query = self.rfile.read(length)
- tmp_dict = cgi.parse_qs(query)
- query_dict = dict([(k, v[0]) for (k, v) in tmp_dict.items()\
- if (k.startswith('kmid')\
- or k == 'lastkmidindex' or k =='dcid')])
- query_dict['lastkmidindex'] = int(query_dict['lastkmidindex'])
- self._logger.debug("Query dict: %s", str(query_dict))
- try:
- km = KMHTTPRequestHandler._km
- redirect_url = km._getDataContainerURL(query_dict)
- if redirect_url != None:
- self._logger.debug("Returning URL '%s'...", redirect_url)
- httpanswer = km.web_interface.get_kmurl(redirect_url,
- query_dict['dcid'])
- else:
- self._logger.debug("Returning Error Code 404: \
-DataContainer ID '%s' not found.", query_dict['dcid'])
- httpanswer = km.web_interface.get_updatequery(query_dict)
- except Exception, excep:
- self._logger.warn("Caught exception: %s", excep.message)
- httpanswer = km.web_interface.get_internalerror(excep)
- else:
- httpanswer = HTTPAnswer(400, "Cannot interpret query.")
- return httpanswer
-
- def _do_POST_request_register_km(self):
- length = int( self.headers['content-length'] )
- query = self.rfile.read(length)
- dict = cgi.parse_qs(query)
- km = KMHTTPRequestHandler._km
- host = dict['remote_km_host'][0]
- port = int(dict['remote_km_port'][0])
- try:
- km.registerKnowledgeManager(host, port, False)
- except KnowledgeManagerException:
- emsg = "Host '%s:%d' is not a KnowledgeManager."
- return HTTPAnswer(400, emsg % (host, port))
- return km.web_interface.get_frontpage("/")
-
- def do_GET(self):
- """
- Returns a requested HDF5 from temporary directory or the web frontend
- if the given request path was located outside the servers tmp directory.
- """
- log = self._logger
- km = KMHTTPRequestHandler._km
- if self.path == '/' or self.path.startswith('/../') or \
- self.path.startswith('/?'):
- km.web_interface.get_frontpage(self.path).sendTo(self)
- elif self.path.startswith(HTTP_REQUEST_DC_DETAILS_PATH):
- km.web_interface.get_details(self.path).sendTo(self)
- elif self.path.startswith(HTTP_REQUEST_SEARCH):
- km.web_interface.get_search(self.path).sendTo(self)
- else:
- f = self.send_head()
- if f:
- self.copyfile(f, self.wfile)
- f.close()
- try:
- log.debug("Trying to remove temporary file '%s'..", f.name)
- os.remove(f.name)
- except Exception:
- log.warn("Cannot delete temporary file '%s'.", f.name)
-
- def send_head(self): # see SimpleHTTPServer.SimpleHTTPRequestHandler
- """
- Sends HTTP headers for HDF5 file requests.
- """
- log = self._logger
- km = KMHTTPRequestHandler._km
- source_dir = km._http_dir # this is intended
- log.debug("HTTP GET request: Reading files from directory '%s'..",
- source_dir)
- try:
- # build filename, remove preceding '/' in path
- filename = os.path.join(source_dir, self.path[1:])
- log.debug("Returning file '%s' as answer for HTTP request..",
- filename)
- f = open(filename, 'rb')
- except IOError:
- self.send_error(404, "File not found")
- return None
- self.send_response(200)
- #self.send_header("Content-type", "application/x-hdf")
- fs = os.fstat(f.fileno())
- self.send_header("Content-Length", str(fs[6]))
- self.send_header("Last-Modified", self.date_time_string(fs.st_mtime))
- self.end_headers()
- return f
-
-
-def _enableLogging():
- """
- Enables logging to stdout for debug purposes.
- """
- l = logging.getLogger("pyphant")
- l.setLevel(logging.DEBUG)
- f = logging.Formatter('%(asctime)s [%(name)s|%(levelname)s] %(message)s')
- h = logging.StreamHandler(sys.stderr)
- h.setFormatter(f)
- l.addHandler(h)
- l.info("Logger 'pyphant' has been configured for debug purposes.")
Added: trunk/src/pyphant/pyphant/core/KnowledgeNode.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeNode.py (rev 0)
+++ trunk/src/pyphant/pyphant/core/KnowledgeNode.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -0,0 +1,411 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# * Neither the name of the Freiburg Materials Research Center,
+# University of Freiburg nor the names of its contributors may be used to
+# endorse or promote products derived from this software without specific
+# prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
+# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+from __future__ import with_statement
+
+"""
+This module provides the KnowledgeNode class which is used as an
+HTTP communication channel between one local KnowledgeManager and
+arbitrary many remote KnowledgeManagers. It comes with a RoutingHTTPServer
+and an optional WebInterface.
+"""
+
+__id__ = "$Id$"
+__author__ = "$Author$"
+__version__ = "$Revision$"
+# $Source: $
+
+from pyphant.core.RoutingHTTPServer import (RoutingHTTPServer,
+ UnreachableError)
+import sqlite3
+from pyphant.core.Helpers import getPyphantPath
+from pyphant.core.SQLiteWrapper import create_table
+from time import time
+from urllib2 import (urlopen, URLError, HTTPError)
+from urllib import urlencode
+import logging
+from pyphant.core.KnowledgeManager import (DCNotFoundError, KnowledgeManager)
+from pyphant.core.bottle import (request, send_file)
+try:
+ from json import (dumps, load, loads)
+except ImportError:
+ from simplejson import (dumps, load, loads)
+from tempfile import (mkdtemp, mkstemp)
+import os
+from pyphant import __path__ as pyphant_source_path
+import pyphant.core.bottle
+
+
+class SkipError(Exception):
+ pass
+
+
+class RemoteError(Exception):
+ pass
+
+
+class RemoteKN(object):
+ """
+ This class represents a remote KnowledgeNode.
+ """
+
+ status_dict = {0:'offline', 1:'online', 2:'disabled'}
+
+ def __init__(self, host, port, status=1, timeout=300.0):
+ """
+ Arguments:
+ - `host`: hostname
+ - `port`: port
+ - `status`: 0: offline, may get online after timeout
+ 1: online, may get offline anytime
+ 2: disabled, use enable() to enable
+ - `timeout`: refresh interval for status lookup when offline
+ default: 5 min
+ """
+ self.host = host
+ self.port = port
+ self.url = "http://%s:%d/" % (host, port)
+ self.timeout = timeout
+ self.last_update = None
+ self.uuid = None
+ self._status = status
+ self.logger = logging.getLogger('pyphant')
+ self.update_status()
+
+ def __eq__(self, other):
+ if not isinstance(other, RemoteKN):
+ return False
+ else:
+ return self.host == other.host and self.port == other.port
+
+ def _get_status(self):
+ return self.status_dict[self._status]
+ status = property(_get_status)
+
+ def enable(self):
+ self._status = 0
+ self.update_status()
+
+ def disable(self):
+ self.last_update = None
+ self.uuid = None
+ self._status = 2
+
+ def update_status(self):
+ if self._status == 2:
+ return
+ elif self.last_update == None or self._status == 1:
+ self.connect()
+ else:
+ if time() - self.last_update > self.timeout:
+ self.connect()
+
+ def connect(self):
+ stream = None
+ try:
+ try:
+ stream = urlopen(self.url + 'uuid/', timeout=3.0)
+ except TypeError:
+ stream = urlopen(self.url + 'uuid/')
+ line = stream.readline()
+ if line.startswith('urn:uuid:'):
+ self._status = 1
+ self.uuid = line
+ else:
+ self._status = 0
+ self.logger.error("Remote KM '%s' returned broken uuid: '%s'" \
+ % (self.url, line))
+ except (URLError, IOError, HTTPError):
+ self._status = 0
+ self.logger.warn("Remote KM '%s' is not responding." % self.url)
+ finally:
+ if stream != None:
+ stream.close()
+ self.last_update = time()
+
+ def get_datacontainer_url(self, dc_id, skip):
+ self.update_status()
+ if self._status == 1:
+ if self.uuid in skip:
+ raise SkipError()
+ else:
+ try:
+ query = urlencode({'skip':dumps(skip), 'dc_id':dc_id})
+ url = '%sget_dc_url/?%s' % (self.url, query)
+ try:
+ stream = urlopen(url, timeout=60.0)
+ except TypeError:
+ stream = urlopen(url)
+ assert stream.headers.type == 'application/json'
+ answer = load(stream)
+ stream.close()
+ if answer['dc_url'] == None:
+ raise DCNotFoundError
+ assert len(answer['skip']) >= len(skip)
+ return answer['dc_url'], answer['skip']
+ except (URLError, HTTPError, IOError, AssertionError):
+ raise UnreachableError()
+ else:
+ raise UnreachableError()
+
+
+class KnowledgeNode(RoutingHTTPServer):
+ """
+ This class manages communication between one local and arbitrary many
+ remote KM instances.
+ """
+
+ def __init__(self, local_km=None,
+ host=u'127.0.0.1', port=8080, start=False,
+ web_interface=False, dbase=u'default'):
+ """
+ Arguments:
+ - `local_km`: Local KnowledgeManager instance to hook up to.
+ If set to `None`, KnowledgeManager.getInstance() is used.
+ - `host`: hostname to listen on
+ - `port`: port to listen on
+ - `start`: flag that indicates whether to start the server
+ - `web_interface`: flag that indicates whether to enable
+ the web interface. You can enable/disable it anytime by
+ setting (KN instance).web_interface.enabled to `True`/`False`.
+ - `dbase`: leave this to 'default', other values are allowed for
+ debug purposes
+ """
+ RoutingHTTPServer.__init__(self, host, port, start)
+ if local_km == None:
+ local_km = KnowledgeManager.getInstance()
+ self.km = local_km
+ self.remotes = []
+ if dbase == u'default':
+ self._dbase = getPyphantPath('/sqlite3/') + 'kn_remotes.sqlite3'
+ else:
+ self._dbase = dbase
+ self._restore_remotes()
+ self._setup_routes()
+ self._tempdir = mkdtemp(prefix = 'HDF5Wrap')
+ tpl_path = pyphant_source_path[0] + '/web/templates/'
+ if not tpl_path in pyphant.core.bottle.TEMPLATE_PATH:
+ pyphant.core.bottle.TEMPLATE_PATH.append(tpl_path)
+ from pyphant.core.WebInterface import WebInterface
+ self.web_interface = WebInterface(self, web_interface)
+ self.km.node = self
+
+ def _restore_remotes(self):
+ """
+ Loads remotes from dbase.
+ """
+ connection = sqlite3.connect(self._dbase)
+ cursor = connection.cursor()
+ try:
+ columns = [('host', 'TEXT'), ('port', 'INT'), ('status', 'INT'),
+ ('', 'UNIQUE(host, port)')]
+ create_table('kn_remotes', columns, cursor)
+ cursor.execute("SELECT * FROM kn_remotes")
+ self.remotes = [RemoteKN(host, port, status) \
+ for host, port, status in cursor]
+ connection.commit()
+ finally:
+ cursor.close()
+ connection.close()
+
+ def _setup_routes(self):
+ self.app.add_route('/uuid/', self.get_uuid)
+ self.app.add_route('/get_dc_url/', self.handle_datacontainer_url)
+ self.app.add_route(r'/wrapped/:filename#..*\.hdf$#',
+ self.handle_wrapped)
+
+ def stop(self):
+ RoutingHTTPServer.stop(self)
+ if not hasattr(self, '_tempdir'):
+ return
+ if os.path.isdir(self._tempdir):
+ from shutil import rmtree
+ try:
+ rmtree(self._tempdir)
+ except OSError:
+ km.logger.warn("Could not delete '%s'." % self._tempdir)
+
+ def register_remote(self, host, port):
+ host = host.lower()
+ port = int(port)
+ connection = sqlite3.connect(self._dbase)
+ cursor = connection.cursor()
+ error = None
+ try:
+ try:
+ cursor.execute("INSERT OR ABORT INTO kn_remotes "\
+ "(host, port, status) "\
+ "VALUES (?, ?, ?)", (host, port, 0))
+ self.remotes.append(RemoteKN(host, port))
+ except sqlite3.IntegrityError:
+ error = RemoteError("Remote '%s:%d' already registered." \
+ % (host, port))
+ connection.commit()
+ finally:
+ cursor.close()
+ connection.close()
+ if not error is None:
+ raise error
+
+ def remove_remote(self, host, port):
+ host = host.lower()
+ port = int(port)
+ dummy = RemoteKN(host, port, status=2)
+ try:
+ self.remotes.remove(dummy)
+ except ValueError:
+ raise RemoteError("Remote '%s:%d' is not registered." \
+ % (host, port))
+ connection = sqlite3.connect(self._dbase)
+ cursor = connection.cursor()
+ try:
+ cursor.execute("DELETE FROM kn_remotes "\
+ "WHERE host=? AND port=?", (host, port))
+ connection.commit()
+ finally:
+ cursor.close()
+ connection.close()
+
+ def change_remote(self, host, port, status):
+ host = host.lower()
+ port = int(port)
+ dummy = RemoteKN(host, port, status=2)
+ for rem in self.remotes:
+ if rem == dummy and rem._status != status:
+ if status == 2:
+ rem.disable()
+ else:
+ rem.enable()
+ connection = sqlite3.connect(self._dbase)
+ cursor = connection.cursor()
+ try:
+ cursor.execute("UPDATE kn_remotes SET status=? "\
+ "WHERE host=? AND port=?",
+ (status, host, port))
+ connection.commit()
+ finally:
+ cursor.close()
+ connection.close()
+ return
+ raise RemoteError("Remote '%s:%d' is not registered." \
+ % (host, port))
+
+ def disable_remote(self, host, port):
+ self.change_remote(host, port, 2)
+
+ def enable_remote(self, host, port):
+ self.change_remote(host, port, 0)
+
+ def get_uuid(self):
+ return self.km.uuid
+ uuid = property(get_uuid)
+
+ def get_datacontainer(self, dc_id):
+ skip = [self.uuid]
+ for remote in self.remotes:
+ try:
+ dc_url, skip = remote.get_datacontainer_url(dc_id, skip)
+ self.km.registerURL(dc_url)
+ return self.km.getDataContainer(dc_id)
+ except (DCNotFoundError, UnreachableError, SkipError):
+ pass
+ raise DCNotFoundError()
+
+ def handle_datacontainer_url(self):
+ query = request.GET
+ skip = loads(query['skip'])
+ if self.uuid in skip:
+ # This should not happen during normal operation
+ self.km.logger.error(
+ "KN '%s' has been queried although it is in the skip list.")
+ else:
+ skip.append(self.uuid)
+ dc_id = query['dc_id']
+ try:
+ dc = self.km.getDataContainer(dc_id, try_remote=False)
+ # Wrap data container in temporary HDF5 file
+ osFileId, filename = mkstemp(suffix='.hdf',
+ prefix='dcrequest-',
+ dir=self._tempdir)
+ os.close(osFileId)
+ handler = self.km.getH5FileHandler(filename, 'w')
+ with handler:
+ handler.saveDataContainer(dc)
+ dc_url = self.url + "wrapped/" + os.path.basename(filename)
+ except DCNotFoundError:
+ dc_url = None
+ for remote in self.remotes:
+ try:
+ dc_url, skip = remote.get_datacontainer_url(dc_id, skip)
+ break
+ except (DCNotFoundError, UnreachableError, SkipError):
+ pass
+ return {'skip':skip, 'dc_url':dc_url}
+
+ def handle_wrapped(self, filename):
+ send_file(filename, self._tempdir,
+ guessmime=False, mimetype='application/x-hdf')
+
+
+def get_kn_autoport(ports, logger=None, *args, **kargs):
+ """
+ Returns a KnowledgeNode listening on the first free port in `ports`
+ messages are logged to `logger` or stdout if `None`
+ If no port is free, a socket.error (no. 98) is raised.
+ """
+ import socket
+ def log(text):
+ if logger is None:
+ print text
+ else:
+ logger.warn(text)
+
+ last_error = None
+ for port in ports:
+ try:
+ kn = KnowledgeNode(port=port, *args, **kargs)
+ return kn
+ except socket.error, err:
+ last_error = err
+ try:
+ #Python 2.6
+ eno = err.errno
+ except AttributeError:
+ #Python 2.5
+ eno = err.args[0]
+ from errno import (EADDRINUSE, EACCES)
+ if eno == EADDRINUSE:
+ log("Port %d is already in use." % port)
+ elif eno == EACCES:
+ log("Port %d: Permission denied." % port)
+ else:
+ raise err
+ raise last_error
Modified: trunk/src/pyphant/pyphant/core/PyTablesPersister.py
===================================================================
--- trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-01-11 21:39:34 UTC (rev 662)
+++ trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -105,20 +105,20 @@
input.flush()
orderGroup._v_attrs.resultPlug = order[1]
-def saveRecipeToHDF5File( recipe, filename ):
+def saveRecipeToHDF5File(recipe, filename, saveResults=True):
_logger.info( "Saving to %s" % filename )
h5 = tables.openFile(filename, 'w')
recipeGroup = h5.createGroup("/", "recipe")
resultsGroup = h5.createGroup("/", "results")
workers=recipe.getWorkers()
for worker in workers:
- saveWorker(h5, recipeGroup, worker)
+ saveWorker(h5, recipeGroup, worker, saveResults)
h5.close()
-def saveWorker(h5, recipeGroup, worker):
+def saveWorker(h5, recipeGroup, worker, saveResults=True):
workerGroup = h5.createGroup(recipeGroup, "worker_"+str(hash(worker)))
saveBaseAttributes(h5, workerGroup, worker)
- savePlugs(h5, workerGroup, worker)
+ savePlugs(h5, workerGroup, worker, saveResults)
saveParameters(h5, workerGroup, worker)
def saveParameters(h5, workerGroup, worker):
@@ -126,11 +126,11 @@
for (paramName, param) in worker._params.iteritems():
h5.setNodeAttr(paramGroup, paramName, param.value)
-def savePlugs(h5, workerGroup, worker):
+def savePlugs(h5, workerGroup, worker, saveResults=True):
plugs = h5.createGroup(workerGroup, "plugs")
for (plugName, plug) in worker._plugs.iteritems():
plugGroup = h5.createGroup(plugs, plugName)
- if plug.resultIsAvailable():
+ if plug.resultIsAvailable() and saveResults:
resId = saveResult(plug._result, h5)
h5.setNodeAttr(plugGroup, "result", resId)
connectionTable = h5.createTable(plugGroup, 'connections', Connection, expectedrows=len(plug._sockets))
Added: trunk/src/pyphant/pyphant/core/RoutingHTTPServer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/RoutingHTTPServer.py (rev 0)
+++ trunk/src/pyphant/pyphant/core/RoutingHTTPServer.py 2010-02-01 17:45:00 UTC (rev 663)
@@ -0,0 +1,184 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 2006-2008, Rectorate of the University of Freiburg
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that ...
[truncated message content] |
|
From: <zk...@us...> - 2010-01-11 21:39:53
|
Revision: 662
http://pyphant.svn.sourceforge.net/pyphant/?rev=662&view=rev
Author: zklaus
Date: 2010-01-11 21:39:34 +0000 (Mon, 11 Jan 2010)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Clutch: Ugly hack to allow for creator attributes in fmf files.
Fix: Fixes a bug in an error message.
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/PyTablesPersister.py
trunk/src/workers/fmfile/fmfile/FMFLoader.py
Modified: trunk/src/pyphant/pyphant/core/PyTablesPersister.py
===================================================================
--- trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2009-12-15 18:20:40 UTC (rev 661)
+++ trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2010-01-11 21:39:34 UTC (rev 662)
@@ -172,7 +172,9 @@
h5.setNodeAttr(resultGroup, "machine", result.machine.encode("utf-8"))
for key,value in result.attributes.iteritems():
if key in _reservedAttributes:
- raise ValueError, "Attribute should not be named %s!" % _reservedAttributes
+ raise ValueError("Attributes should not be named %s, "
+ "but one was in fact called %s!"
+ % (str(_reservedAttributes), key))
h5.setNodeAttr(resultGroup,key,value)
#Store fields of sample Container and gather list of field IDs
columns = []
Modified: trunk/src/workers/fmfile/fmfile/FMFLoader.py
===================================================================
--- trunk/src/workers/fmfile/fmfile/FMFLoader.py 2009-12-15 18:20:40 UTC (rev 661)
+++ trunk/src/workers/fmfile/fmfile/FMFLoader.py 2010-01-11 21:39:34 UTC (rev 662)
@@ -242,7 +242,14 @@
newField.dimensions[dim]=independentFields[indepField]
assert newField.isValid()
containers.append(newField)
- result = DataContainer.SampleContainer(containers,attributes=commonAttr)
+ #The next lines are a hack and should be dealt with properly...
+ if u'creator' in commonAttr.keys():
+ creator = commonAttr[u'creator']
+ del commonAttr[u'creator']
+ result = DataContainer.SampleContainer(containers,attributes=commonAttr)
+ result.creator = creator
+ else:
+ result = DataContainer.SampleContainer(containers,attributes=commonAttr)
return result
def reshapeField(field):
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2009-12-15 18:20:50
|
Revision: 661
http://pyphant.svn.sourceforge.net/pyphant/?rev=661&view=rev
Author: zklaus
Date: 2009-12-15 18:20:40 +0000 (Tue, 15 Dec 2009)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Enh: Added proper BOM treatment to FMFLoader.py
Modified Paths:
--------------
trunk/src/workers/fmfile/fmfile/FMFLoader.py
Modified: trunk/src/workers/fmfile/fmfile/FMFLoader.py
===================================================================
--- trunk/src/workers/fmfile/fmfile/FMFLoader.py 2009-11-23 13:49:19 UTC (rev 660)
+++ trunk/src/workers/fmfile/fmfile/FMFLoader.py 2009-12-15 18:20:40 UTC (rev 661)
@@ -40,6 +40,7 @@
# $Source$
import zipfile, numpy, re, collections, copy, StringIO, os.path
+import codecs
from pyphant.core import (Worker, Connectors,
Param, DataContainer)
from pyphant.quantities.PhysicalQuantities import PhysicalQuantity,isPhysicalUnit,isPhysicalQuantity
@@ -460,6 +461,8 @@
localVar = {'fmf-version':'1.0','coding':'utf-8',
'delimiter':'\t'}
commentChar = ';'
+ if b.startswith(codecs.BOM_UTF8):
+ b = b.lstrip(codecs.BOM_UTF8)
if b[0] == ';' or b[0] == '#':
commentChar = b[0]
items = [var.strip().split(':') for var in b.split('-*-')[1].split(';')]
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <zk...@us...> - 2009-11-23 13:49:30
|
Revision: 660
http://pyphant.svn.sourceforge.net/pyphant/?rev=660&view=rev
Author: zklaus
Date: 2009-11-23 13:49:19 +0000 (Mon, 23 Nov 2009)
Log Message:
-----------
Merge branch 'master' into svn-trunk
Modified Paths:
--------------
trunk/src/pyphant/pyphant/core/DataContainer.py
trunk/src/pyphant/pyphant/core/FieldContainer.py
trunk/src/pyphant/pyphant/core/H5FileHandler.py
trunk/src/pyphant/pyphant/core/Helpers.py
trunk/src/pyphant/pyphant/core/KnowledgeManager.py
trunk/src/pyphant/pyphant/core/PyTablesPersister.py
trunk/src/pyphant/pyphant/core/WebInterface.py
trunk/src/pyphant/pyphant/quantities/PhysicalQuantities.py
trunk/src/pyphant/pyphant/tests/TestAutoFocus.py
trunk/src/pyphant/pyphant/tests/TestH5FileHandler.py
trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py
trunk/src/pyphant/pyphant/tests/TestPyTablesPersister.py
trunk/src/pyphant/pyphant/wxgui2/wxPyphantApplication.py
trunk/src/workers/ImageProcessing/ImageProcessing/AutoFocus.py
trunk/src/workers/ImageProcessing/ImageProcessing/MeasureFocus.py
trunk/src/workers/ImageProcessing/ImageProcessing/NDImageWorker.py
trunk/src/workers/ImageProcessing/ImageProcessing/Watershed.py
trunk/src/workers/tools/tools/Emd5Src.py
Added Paths:
-----------
trunk/src/pyphant/pyphant/core/SQLiteWrapper.py
trunk/src/pyphant/pyphant/tests/TestSQLiteWrapper.py
Modified: trunk/src/pyphant/pyphant/core/DataContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/DataContainer.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/DataContainer.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -229,6 +229,10 @@
return u"%s %s" % (self.longname, self.shortname)
label = property(_getLabel)
+ def _getRawDataBytes(self):
+ return [column.rawDataBytes for column in self.columns]
+ rawDataBytes = property(_getRawDataBytes)
+
def generateHash(self, m=None):
if m == None:
m = hashlib.md5()
Modified: trunk/src/pyphant/pyphant/core/FieldContainer.py
===================================================================
--- trunk/src/pyphant/pyphant/core/FieldContainer.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/FieldContainer.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -61,6 +61,7 @@
import os, platform, datetime, socket, urlparse
from pyphant.quantities.PhysicalQuantities import (isPhysicalQuantity, PhysicalQuantity,_prefixes)
from DataContainer import DataContainer, enc, _logger
+from types import NoneType
#Default variables of indices
INDEX_NAMES=[u'i', u'j', u'k', u'l', u'm', u'n']
@@ -74,6 +75,7 @@
hash = hashlib.md5().hexdigest()
shortname=u"i"
longname=u"index"
+ rawDataBytes = 0
def seal(self, id=None):
pass
@@ -260,6 +262,11 @@
return label.replace('1.0 ',r'')#.replace('mu',u'\\textmu{}')
shortlabel=property(_getShortLabel)
+ def _getRawDataBytes(self):
+ return self.data.nbytes \
+ + sum([dim.rawDataBytes for dim in self.dimensions])
+ rawDataBytes = property(_getRawDataBytes)
+
def __deepcopy__(self, memo):
self.lock.acquire()
data=copy.deepcopy(self.data, memo)
@@ -353,7 +360,7 @@
def __eq__(self, other, rtol=1e-5, atol=1e-8):
if type(self) != type(other):
- if type(other) != IndexMarker:
+ if type(other) != IndexMarker and type(other) != NoneType:
_logger.debug('Cannot compare objects with different type (%s and %s).' % (type(self),type(other)))
return False
if not (self.typeString == other.typeString):
Modified: trunk/src/pyphant/pyphant/core/H5FileHandler.py
===================================================================
--- trunk/src/pyphant/pyphant/core/H5FileHandler.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/H5FileHandler.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -51,9 +51,9 @@
"""
This class is used to handle IO operations on HDF5 files.
"""
- def __init__(self, filename, mode = 'a'):
+ def __init__(self, filename, mode='a'):
"""
- Opens a HDF5 file.
+ Opens an HDF5 file.
filename -- path to the file that should be opened
mode -- mode in which file is opened. Possible values: 'r', 'w', 'a'
meaning 'read only', 'overwrite' and 'append'.
@@ -98,6 +98,19 @@
uriType = uriType.encode('utf-8')
return (resNode, uriType)
+ def isIndexMarker(self, dcId):
+ """
+ returns True iff the underlying HDF5 file contains dcId and
+ dcId belongs to an IndexMarker
+ """
+ resNode, uriType = self.getNodeAndTypeFromId(dcId)
+ if uriType == u'field':
+ try:
+ resNode._g_checkHasChild('dimensions')
+ except tables.NoSuchNodeError:
+ return True
+ return False
+
def loadDataContainer(self, dcId):
"""
Loads a DataContainer from the HDF5 file and returns it as a
@@ -106,13 +119,9 @@
"""
resNode, uriType = self.getNodeAndTypeFromId(dcId)
if uriType == 'field':
- _logger.info("Trying to load field data from node %s..." % resNode)
result = self.loadField(resNode)
- _logger.info("...successfully loaded.")
elif uriType == 'sample':
- _logger.info("Trying to load sample data from node %s..." % resNode)
result = self.loadSample(resNode)
- _logger.info("...successfully loaded.")
else:
raise TypeError, "Unknown result uriType in <%s>" % resNode._v_title
return result
@@ -137,59 +146,52 @@
"""
Extracts meta data about a given DataContainer and returns it
as a dictionary.
- dcId -- emd5 of the DC to summarize. If dcId == None, a dictionary
- that maps emd5s to summaries is returned.
+ dcId -- emd5 of the DC to summarize. If the emd5 belongs to an
+ IndexMarker object, u'IndexMarker' is returned.
+ If dcId == None, a dictionary that maps emd5s to summaries
+ is returned, where IndexMarker objects are ignored.
"""
if dcId == None:
summary = {}
for group in self.handle.walkGroups(where = "/results"):
currDcId = group._v_attrs.TITLE
if len(currDcId) > 0:
- summary[currDcId] = self.loadSummary(currDcId)
+ tmp = self.loadSummary(currDcId)
+ if tmp != u'IndexMarker':
+ summary[currDcId] = tmp
+ elif self.isIndexMarker(dcId):
+ return u'IndexMarker'
else:
+ from pyphant.core.Helpers import (utf82uc, emd52dict)
summary = {}
summary['id'] = dcId
resNode, uriType = self.getNodeAndTypeFromId(dcId)
- summary['longname'] = unicode(self.handle.getNodeAttr(resNode,
- "longname"), 'utf-8')
- summary['shortname'] = unicode(self.handle.getNodeAttr(resNode,
- "shortname"), 'utf-8')
- emd5_split = dcId.split('/')
+ summary['longname'] = utf82uc(self.handle.getNodeAttr(resNode,
+ "longname"))
+ summary['shortname'] = utf82uc(self.handle.getNodeAttr(resNode,
+ "shortname"))
+ summary.update(emd52dict(dcId))
try:
- summary['machine'] = unicode(self.handle.getNodeAttr(resNode,
- "machine"), 'utf-8')
- summary['creator'] = unicode(self.handle.getNodeAttr(resNode,
- "creator"), 'utf-8')
+ summary['machine'] = utf82uc(self.handle.getNodeAttr(resNode,
+ "machine"))
+ summary['creator'] = utf82uc(self.handle.getNodeAttr(resNode,
+ "creator"))
except:
- summary['machine'] = unicode(emd5_split[2], 'utf-8')
- summary['creator'] = unicode(emd5_split[3], 'utf-8')
- summary['date'] = unicode(emd5_split[4], 'utf-8')
- summary['hash'] = emd5_split[5].split('.')[0]
- summary['type'] = unicode(emd5_split[5].split('.')[1], 'utf-8')
+ pass # machine, creator set by emd52dict(dcId) before
attributes = {}
if uriType == 'field':
for key in resNode.data._v_attrs._v_attrnamesuser:
attributes[key]=self.handle.getNodeAttr(resNode.data, key)
- unit = eval(unicode(self.handle.getNodeAttr(resNode, "unit"),
- 'utf-8'))
- try:
- if isinstance(unit, (str, unicode)):
- unit = unit.replace('^', '**')
- if isinstance(unit, unicode):
- unit = unit.encode('utf-8')
- summary['unit'] = PhysicalQuantity(unit)
- except:
- try:
- summary['unit'] = PhysicalQuantity("1" + unit)
- except:
- summary['unit'] = unit
- try:
- dimTable = resNode.dimensions
- dimensions = [self.loadSummary(row['id'])
+ unit = eval(utf82uc(self.handle.getNodeAttr(resNode, "unit")))
+ summary['unit'] = unit
+ dimTable = resNode.dimensions
+ def filterIndexMarker(emd5):
+ if self.isIndexMarker(emd5):
+ return u'IndexMarker'
+ else:
+ return emd5
+ dimensions = [filterIndexMarker(row['id']) \
for row in dimTable.iterrows()]
- except tables.NoSuchNodeError:
- dimensions = u'INDEX'
- summary['type'] = u'index'
summary['dimensions'] = dimensions
elif uriType == 'sample':
for key in resNode._v_attrs._v_attrnamesuser:
@@ -199,7 +201,7 @@
for resId in self.handle.getNodeAttr(resNode, "columns"):
nodename = "/results/" + resId
columnId = self.handle.getNodeAttr(nodename, "TITLE")
- columns.append(self.loadSummary(columnId))
+ columns.append(columnId)
summary['columns'] = columns
summary['attributes'] = attributes
return summary
Modified: trunk/src/pyphant/pyphant/core/Helpers.py
===================================================================
--- trunk/src/pyphant/pyphant/core/Helpers.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/Helpers.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -116,3 +116,18 @@
if isinstance(stype, ListType):
return map(convert, stype)
return convert(stype)
+
+def emd52dict(emd5):
+ """
+ returns a dictionary with keys
+ ('machine', 'creator', 'date', 'hash', 'type')
+ """
+ emd5 = utf82uc(emd5)
+ emd5_split = emd5.split('/')
+ retdict = {}
+ retdict['machine'] = emd5_split[2]
+ retdict['creator'] = emd5_split[3]
+ retdict['date'] = emd5_split[4]
+ retdict['hash'] = emd5_split[5].split('.')[0]
+ retdict['type'] = emd5_split[5].split('.')[1]
+ return retdict
Modified: trunk/src/pyphant/pyphant/core/KnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/KnowledgeManager.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -60,17 +60,18 @@
KMHTMLParser,
ThreadedHTTPServer)
from fmfile import FMFLoader
+from pyphant.core.SQLiteWrapper import (SQLiteWrapper, AnyValue)
WAITING_SECONDS_HTTP_SERVER_STOP = 5
HTTP_REQUEST_DC_URL_PATH = "/request_dc_url"
HTTP_REQUEST_KM_ID_PATH = "/request_km_id"
HTTP_REQUEST_DC_DETAILS_PATH = "/request_dc_details?dcid="
-# Maximum number of DCs to store in cache:
-CACHE_SIZE = 10
-# Timeout for cached DCs in seconds:
-CACHE_TIMEOUT = 3600
-# Number of hits a DC has to have at least in order to be cached:
-CACHE_THRESHHOLD = 2
+HTTP_REQUEST_REGISTER_KM = "/request_register_km"
+HTTP_REQUEST_SEARCH = "/request_search"
+# Limit for sum(DC.rawDataBytes) for DC in cache:
+CACHE_MAX_SIZE = 256 * 1024 * 1024
+# Limit for number of stored DCs in cache:
+CACHE_MAX_NUMBER = 100
KM_PATH = '/KMstorage/'
REHDF5 = re.compile(r'..*\.h5$|..*\.hdf$|..*\.hdf5$')
REFMF = re.compile(r'..*\.fmf$')
@@ -98,18 +99,23 @@
os.mkdir(makedir)
return path
-def getFilenameFromDcId(dcId):
+def getFilenameFromDcId(dcId, temporary=False):
"""
Returns a unique filename for the given emd5.
"""
emd5list = urlparse(dcId + '.h5')[2][2:].split('/')
emd5path = ''
- for p in emd5list:
+ for p in emd5list[:-2]:
emd5path += (p + '/')
- emd5path = emd5path[:-1]
+ emd5path += emd5list[-2][:10] + '/' + emd5list[-2][11:]\
+ + '.' + emd5list[-1]
directory = os.path.dirname(emd5path)
filename = os.path.basename(emd5path)
- return getPyphantPath(KM_PATH + 'by_emd5/' + directory) + filename
+ if temporary:
+ subdir = 'tmp/by_emd5/'
+ else:
+ subdir = 'by_emd5/'
+ return getPyphantPath(KM_PATH + subdir + directory) + filename
class KnowledgeManagerException(Exception):
@@ -138,6 +144,24 @@
return self._parent_excep
+class CachedDC(object):
+ def __init__(self, dc_ref):
+ self.id = dc_ref.id
+ self.ref = dc_ref
+ self.size = dc_ref.rawDataBytes
+
+ def __eq__(self, other):
+ return self.id == other.id
+
+
+class TestCachedDC(object):
+ def __init__(self, dc_id):
+ self.id = dc_id
+
+ def __eq__(self, other):
+ return self.id == other.id
+
+
class KnowledgeManager(Singleton):
"""
Knowledge Manager for Pyphant
@@ -174,115 +198,74 @@
"""
def __init__(self):
"""
- Sets the unique id for the KM instance and restores all HDF5 files from
- the .pyphant directory.
+ Sets the unique id for the KM instance, sets up the DataBase if
+ it has not been initialized yet and clears the tmp dir.
"""
super(KnowledgeManager, self).__init__()
self._logger = logging.getLogger("pyphant")
- self._storage = {}
- self._cache = {}
+ self._cache = []
+ self._cache_size = 0
self.H5FileHandlers = {}
self._remoteKMs = {} # key:id, value:url
self._server = None
self._server_id = uuid1()
- self.restoreKnowledge()
self.web_interface = WebInterface(self, True)
+ self.dbase = getPyphantPath('/sqlite3/') + "km_meta.sqlite3"
+ self.any_value = AnyValue()
+ with SQLiteWrapper(self.dbase) as wrapper:
+ wrapper.setup_dbase()
+ tmpdir = getPyphantPath(KM_PATH + 'tmp/')
+ if os.path.isdir(tmpdir):
+ from shutil import rmtree
+ try:
+ rmtree(tmpdir)
+ except OSError:
+ print "Could not delete '%s'." % (tmpdir, )
- def __del__(self):
+ def tearDown(self):
"""
- Stops the HTTP server and closes all open files.
+ Stops the HTTP server
"""
if self.isServerRunning():
self.stopServer()
- def registerH5(self, filename, mode = 'a', registerContents = True):
+ def hasDataContainer(self, dcid):
"""
- Adds the given HDF5 file to the pool and handles all
- further IO operations on the given file in a save way. If you want the
- knowledge to be stored permanently, use registerURL.
- Possible usage: Register a file that does not exist (setting
- registerContents to False). It then is created.
- Get its H5FileHandler using the getH5FileHandler method.
- Use this FileHandler to write some Data to the file.
- Use the refreshH5 method to import the new contents into the
- KnowledgeManager.
- filename -- path to the HDF5 file to be registered.
- mode -- see H5FileHandler
- registerContents -- whether to register contents of the file as well.
+ Returns whether the given DC is stored locally.
+ Never use this method in a 'with SQLiteWrapper(...) as wrapper'
+ statement! Use wrapper.has_entry(dcid) instead if you already
+ have a wrapper at your hands or you may end up in a sqlite3 locking
+ loop.
"""
- if self.H5FileHandlers.has_key(filename):
- raise KnowledgeManagerException("'%s' has already been registered."
- % filename)
- self.H5FileHandlers[filename] = H5FileHandler(filename, mode)
- if registerContents:
- self.refreshH5(filename)
+ with SQLiteWrapper(self.dbase) as wrapper:
+ has_entry = wrapper.has_entry(dcid)
+ return has_entry
- def getH5FileHandler(self, filename):
+ def getH5FileHandler(self, filename, mode='r'):
"""
- Returns a H5FileHandler for the given filename to perform IO
- operations on the file in a save way. The file has
- to be registered first using registerH5.
- As soon as you are done with your IO operations, use refreshH5
- in order to update the changes.
+ Returns an H5FileHandler for the given filename to perform IO
+ operations on the file in a save way.
filename -- path to the HDF5 file
+ mode -- see H5FileHandler
"""
- if self.H5FileHandlers.has_key(filename):
- return self.H5FileHandlers[filename]
- else:
- raise KnowledgeManagerException("'%s' has not been registered.")
+ return H5FileHandler(filename, mode)
- def refreshH5(self, filename):
+ def registerH5(self, filename, temporary=False):
"""
- Refreshes the contents of the given file. The file has to be
- registered first using registerH5. If a DC emd5 is found that is
- already known to the KnownledgeManager, it is not updated, following
- the principle that emd5s are unique and DCs that have been given a emd5
- should not me modified any more in any way.
+ Adds the given file to the knowledge pool. If you want the data to
+ be copied to the .pyphant directory, use registerURL() instead.
filename -- path to the HDF5 file
+ temporary -- flag that marks data to be deleted upon next
+ instantiation of a KM Singleton
"""
h5fh = self.getH5FileHandler(filename)
with h5fh:
summaryDict = h5fh.loadSummary()
- for dcId, summary in summaryDict.items():
- if not self._storage.has_key(dcId):
- self._storage[dcId] = {'lasthit':None,
- 'hitcount':0,
- 'filehandler':h5fh,
- 'summary':summary}
+ with SQLiteWrapper(self.dbase) as wrapper:
+ for dcId, summary in summaryDict.items():
+ if not wrapper.has_entry(dcId):
+ wrapper.set_entry(summary, filename, temporary)
- def restoreKnowledge(self):
- """
- Restores knowledge from pyphant path.
- """
- def walkfiles(arg, dirname, fnames):
- for fname in fnames:
- if REHDF5.match(fname.lower()) != None:
- if dirname.endswith('/'):
- dirname = dirname[:-1]
- filename = dirname + '/' + fname
- try:
- self.registerH5(filename)
- except Exception:
- self._logger.warn("Could not import '%s'.", filename)
- os.path.walk(getPyphantPath(KM_PATH), walkfiles, None)
-
- def getSummary(self, dcId = None):
- """
- Behaves like H5FileHandler.loadSummary(dcId) except that for
- dcId == None all DataContainers that are stored locally are browsed.
- """
- if dcId == None:
- summary = {}
- for emd5, dcInfo in self._storage.iteritems():
- summary[emd5] = dcInfo['summary']
- else:
- if not self._storage.has_key(dcId):
- raise KnowledgeManagerException("DC with ID '%s' is unknown."\
- % (dcId, ))
- dcInfo = self._storage[dcId]
- summary = dcInfo['summary']
- return summary
-
def _getServerURL(self):
"""
Returns the URL of the HTTP server.
@@ -409,17 +392,22 @@
% (km_url, ), excep)
self._remoteKMs[km_id] = km_url
- def registerURL(self, url):
+ def registerURL(self, url, temporary=False):
"""
- Registers an HDF5 or FMF file downloadable from given URL and store it
- permanently in the .pyphant directory. The content of the file is made
+ Registers an HDF5 or FMF file downloadable from given URL and stores it
+ in the .pyphant directory. The content of the file is made
available to the KnowledgeManager.
HTTP redirects are resolved. The filetype is determined by the
extension.
url -- URL of the HDF5 or FMF file
+ temporary -- set to True in order to mark the data to be deleted upon
+ next instantiation of a KM singleton
"""
parsed = urlparse(url)
- filename = KM_PATH + 'registered/' + parsed[1] + '/'
+ tmp_extension = ''
+ if temporary:
+ tmp_extension = 'tmp/'
+ filename = KM_PATH + tmp_extension + 'registered/' + parsed[1] + '/'
filename += os.path.basename(parsed[2])
directory = os.path.dirname(filename)
filename = getPyphantPath(directory) + os.path.basename(filename)
@@ -445,43 +433,47 @@
savedto, headers = urllib.urlretrieve(url, filename)
self._logger.info("Header information: %s", (str(headers), ))
if REFMF.match(filename.lower()) != None:
- self.registerFMF(filename)
+ self.registerFMF(filename, temporary)
elif REHDF5.match(filename.lower()) != None:
- self.registerH5(filename)
+ self.registerH5(filename, temporary)
else:
raise KnowledgeManagerException('Filetype unknown: %s'
% (filename, ))
- def registerDataContainer(self, dc):
+ def registerDataContainer(self, dc, temporary=False):
"""
Registers a DataContainer located in memory using a given
- reference and store it permanently.
+ reference and stores it in the pyphant directory.
The DataContainer must have an .id attribute,
which could be generated by the datacontainer.seal() method.
If the DCs emd5 is already known to the KnowledgeManager,
the DC is not registered again since emd5s are unique.
dc -- reference to the DataContainer object
+ temporary -- dc is stored only until another KM singleton is
+ created. Set this flag to True e.g. for unit tests
+ or whenever you do not want to produce garbage on
+ your hard drive.
"""
if dc.id == None:
raise KnowledgeManagerException("Invalid id for DataContainer '"\
+ dc.longname + "'")
- if not self._storage.has_key(dc.id):
- filename = getFilenameFromDcId(dc.id)
- self.registerH5(filename, 'w', False)
- handler = self.getH5FileHandler(filename)
+ if not self.hasDataContainer(dc.id):
+ filename = getFilenameFromDcId(dc.id, temporary)
+ handler = self.getH5FileHandler(filename, 'w')
with handler:
handler.saveDataContainer(dc)
- self.refreshH5(filename)
+ self.registerH5(filename, temporary)
- def registerFMF(self, filename):
+ def registerFMF(self, filename, temporary=False):
"""
Extracts a SampleContainer from a given FMF file and stores it
permanently. The emd5 of the SampleContainer that has been generated
is returned.
filename -- path to the FMF file
+ temporary -- see registerDataContainer
"""
sc = FMFLoader.loadFMFFromFile(filename)
- self.registerDataContainer(sc)
+ self.registerDataContainer(sc, temporary)
return sc.id
def _getDCURLFromRemoteKMs(self, query_dict):
@@ -560,7 +552,7 @@
"""
assert self.isServerRunning(), "Server is not running."
dc_id = query_dict['dcid']
- if self._storage.has_key(dc_id):
+ if self.hasDataContainer(dc_id):
dc = self.getDataContainer(dc_id, True, False)
# Wrap data container in temporary HDF5 file
osFileId, filename = tempfile.mkstemp(suffix = '.h5',
@@ -579,6 +571,41 @@
"URL for DC ID '%s' not found." % (dc_id, ), excep)
return dc_url
+ def getDCFromCache(self, dc_id, filename):
+ """
+ Returns a DC instance from cache or local storage.
+ Also puts DC to cache if reasonable.
+ fc_id: emd5 to look for in cache
+ filename: alternative source if dc_id not present in cache
+ """
+ try:
+ index = self._cache.index(TestCachedDC(dc_id))
+ cached = self._cache.pop(index)
+ self._cache.append(cached)
+ return cached.ref
+ except ValueError:
+ with self.getH5FileHandler(filename) as handler:
+ dc = handler.loadDataContainer(dc_id)
+ self._attemptToCacheDC(dc)
+ return dc
+
+ def _attemptToCacheDC(self, dc):
+ cache_item = CachedDC(dc)
+ if cache_item.size > CACHE_MAX_SIZE:
+ return
+ number_fits = len(self._cache) < CACHE_MAX_NUMBER
+ self._cache.reverse()
+ if not number_fits:
+ self._cache_size -= self._cache.pop().size
+ desired_size = CACHE_MAX_SIZE - cache_item.size
+ not_size_fits = self._cache_size > desired_size
+ while not_size_fits:
+ self._cache_size -= self._cache.pop().size
+ not_size_fits = self._cache_size > desired_size
+ self._cache.reverse()
+ self._cache.append(cache_item)
+ self._cache_size += cache_item.size
+
def getDataContainer(self, dc_id, use_cache = True, try_remote = True):
"""
Returns DataContainer matching the given id.
@@ -587,55 +614,18 @@
lookups (default: True)
try_remote -- Try to get DC from remote KMs (default: True)
"""
- dc = None
- found_in_cache = False
- islocal = self._storage.has_key(dc_id)
- if islocal:
- dcinfo = self._storage[dc_id]
- now = time.time()
- if dcinfo['lasthit'] != None:
- if (now - dcinfo['lasthit']) >= CACHE_TIMEOUT:
- dcinfo['hitcount'] = 0
- dcinfo['lasthit'] = now
- dcinfo['hitcount'] += 1
+ filename = None
+ with SQLiteWrapper(self.dbase) as wrapper:
+ try:
+ filename = wrapper[dc_id]['storage']
+ except KeyError:
+ pass
+ if filename != None:
if use_cache:
- if self._cache.has_key(dc_id):
- dc = self._cache[dc_id]
- found_in_cache = True
- self._logger.debug("DC with ID '%s' found in cache.", dc_id)
- else:
- self._logger.debug("DC with ID '%s' not found in cache.",
- dc_id)
- if not found_in_cache:
- try:
- handler = dcinfo['filehandler']
- with handler:
- dc = handler.loadDataContainer(dc_id)
- except Exception, excep:
- raise KnowledgeManagerException("DC ID '%s' known, but "
- "cannot be read"
- "." % (dc_id, ), excep)
- if use_cache and dcinfo['hitcount'] >= CACHE_THRESHHOLD:
- docache = False
- if len(self._cache) >= CACHE_SIZE:
- minhitcount = sys.maxint
- for cachedid in self._cache.keys():
- cacheddcinfo = self._storage[cachedid]
- if (now - cacheddcinfo['lasthit']) >= CACHE_TIMEOUT:
- cacheddcinfo['hitcount'] = 0
- self._cache.pop(cachedid)
- docache = True
- break
- elif cacheddcinfo['hitcount'] < minhitcount:
- minhitcount = cacheddcinfo['hitcount']
- if docache == False \
- and dcinfo['hitcount'] > minhitcount:
- docache == True
- else:
- docache = True
- if docache:
- self._cache[dc_id] = dc
- self._logger.debug("Cached DC with ID '%s'", dc_id)
+ return self.getDCFromCache(dc_id, filename)
+ with self.getH5FileHandler(filename) as handler:
+ dc = handler.loadDataContainer(dc_id)
+ return dc
elif try_remote:
dc_url = self._getDCURLFromRemoteKMs({'dcid':dc_id,
'lastkmidindex':-1})
@@ -645,14 +635,48 @@
filename = getFilenameFromDcId(dc_id)
urllib.urlretrieve(dc_url, filename)
self.registerH5(filename)
- with self.H5FileHandlers[filename] as handler:
+ with self.getH5FileHandler(filename) as handler:
dc = handler.loadDataContainer(dc_id)
+ return dc
else:
raise KnowledgeManagerException("DC ID '%s' is unknown."
% (dc_id, ))
- return dc
+ def getEmd5List(self):
+ """
+ returns a list with all locally known DataContainer ids.
+ """
+ with SQLiteWrapper(self.dbase) as wrapper:
+ return wrapper.get_emd5_list()
+ def search(self, result_keys, search_dict={}, order_by=None,
+ order_asc=True, limit=-1, offset=0, distinct=False):
+ """
+ See SQLiteWrapper.get_andsearch_result()
+ """
+ with SQLiteWrapper(self.dbase) as wrapper:
+ return wrapper.get_andsearch_result(
+ result_keys, search_dict, order_by, order_asc,
+ limit, offset, distinct)
+
+ def getSummary(self, dc_id):
+ """
+ This method returns a dictionary with meta information about
+ the given DC.
+ """
+ with SQLiteWrapper(self.dbase) as wrapper:
+ # TODO: usage of rowwrapper is not optimal in performance
+ rowwrapper = wrapper[dc_id]
+ keys = list(SQLiteWrapper.all_keys)
+ if dc_id.endswith('field'):
+ keys.remove('columns')
+ elif dc_id.endswith('sample'):
+ keys.remove('unit')
+ keys.remove('dimensions')
+ summary = dict([(key, rowwrapper[key]) for key in keys])
+ return summary
+
+
class KMHTTPRequestHandler(SimpleHTTPRequestHandler):
"""
Helper class for KnowledgeManager that handles HTTP requests.
@@ -691,6 +715,8 @@
httpanswer = self._do_POST_request_dc_url()
elif self.path == HTTP_REQUEST_KM_ID_PATH:
httpanswer = self._do_POST_request_km_id()
+ elif self.path == HTTP_REQUEST_REGISTER_KM:
+ httpanswer = self._do_POST_request_register_km()
else:
code = 400
message = "Unknown request path '%s'." % (self.path, )
@@ -753,6 +779,20 @@
httpanswer = HTTPAnswer(400, "Cannot interpret query.")
return httpanswer
+ def _do_POST_request_register_km(self):
+ length = int( self.headers['content-length'] )
+ query = self.rfile.read(length)
+ dict = cgi.parse_qs(query)
+ km = KMHTTPRequestHandler._km
+ host = dict['remote_km_host'][0]
+ port = int(dict['remote_km_port'][0])
+ try:
+ km.registerKnowledgeManager(host, port, False)
+ except KnowledgeManagerException:
+ emsg = "Host '%s:%d' is not a KnowledgeManager."
+ return HTTPAnswer(400, emsg % (host, port))
+ return km.web_interface.get_frontpage("/")
+
def do_GET(self):
"""
Returns a requested HDF5 from temporary directory or the web frontend
@@ -765,6 +805,8 @@
km.web_interface.get_frontpage(self.path).sendTo(self)
elif self.path.startswith(HTTP_REQUEST_DC_DETAILS_PATH):
km.web_interface.get_details(self.path).sendTo(self)
+ elif self.path.startswith(HTTP_REQUEST_SEARCH):
+ km.web_interface.get_search(self.path).sendTo(self)
else:
f = self.send_head()
if f:
@@ -795,7 +837,7 @@
self.send_error(404, "File not found")
return None
self.send_response(200)
- self.send_header("Content-type", "application/x-hdf")
+ #self.send_header("Content-type", "application/x-hdf")
fs = os.fstat(f.fileno())
self.send_header("Content-Length", str(fs[6]))
self.send_header("Last-Modified", self.date_time_string(fs.st_mtime))
Modified: trunk/src/pyphant/pyphant/core/PyTablesPersister.py
===================================================================
--- trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2009-09-28 10:33:55 UTC (rev 659)
+++ trunk/src/pyphant/pyphant/core/PyTablesPersister.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -293,9 +293,10 @@
creator = unicode(h5.getNodeAttr(resNode, "creator"), 'utf-8')
machine = unicode(h5.getNodeAttr(resNode, "machine"), 'utf-8')
except:
- import Helpers
- creator = Helpers.getUsername()
- machine = Helpers.getMachine()
+ from pyphant.core.Helpers import emd52dict
+ emd5dict = emd52dict(resNode._v_title)
+ creator = emd5dict['creator']
+ machine = emd5dict['machine']
data = scipy.array(resNode.data.read())
def loads(inputList):
if type(inputList)==type([]):
Added: trunk/src/pyphant/pyphant/core/SQLiteWrapper.py
===================================================================
--- trunk/src/pyphant/pyphant/core/SQLiteWrapper.py (rev 0)
+++ trunk/src/pyphant/pyphant/core/SQLiteWrapper.py 2009-11-23 13:49:19 UTC (rev 660)
@@ -0,0 +1,672 @@
+# -*- coding: utf-8 -*-
+
+# Copyright (c) 2006-2009, Rectorate of the University of Freiburg
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+# * Neither the name of the Freiburg Materials Research Center,
+# University of Freiburg nor the names of its contributors may be used to
+# endorse or promote products derived from this software without specific
+# prior written permission.
+#
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER
+# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+This module provides a wrapper class that translates the
+KnowledgeManager's summary dictionaries to an SQLite3 database.
+"""
+__id__ = "$Id$"
+__author__ = "$Author$"
+__version__ = "$Revision$"
+# $Source$:
+
+import sqlite3
+import time
+from pyphant.core.Helpers import (utf82uc, uc2utf8, emd52dict)
+from pyphant.quantities.PhysicalQuantities import (PhysicalQuantity,
+ PhysicalUnit)
+from types import (FloatType, IntType, LongType, StringTypes)
+
+def quantity2powers(quantity):
+ if isinstance(quantity, PhysicalQuantity):
+ return tuple(quantity.unit.powers)
+ elif isinstance(quantity, (FloatType, IntType, LongType)):
+ return (0, ) * 10
+ else:
+ raise ValueError("Expected (PhysicalQuantity, FloatType, IntType, "\
+ "LongType) but got %s instead."\
+ % (type(quantity), ))
+
+def str2number(str):
+ try:
+ value = int(str)
+ except ValueError:
+ try:
+ value = long(str)
+ except ValueError:
+ value = float(str)
+ return value
+
+def quantity2dbase(quantity):
+ if isinstance(quantity, (FloatType, IntType, LongType)):
+ return quantity.__repr__()
+ elif isinstance(quantity, PhysicalQuantity):
+ return "P%s;%s" % (quantity.value.__repr__(),
+ quantity.getUnitName())
+ else:
+ raise ValueError("Expected (PhysicalQuantity, FloatType, IntType, "\
+ "LongType) but got %s instead."\
+ % (type(quantity), ))
+
+def dbase2quantity(dbase):
+ if isinstance(dbase, StringTypes):
+ if dbase.startswith("P"):
+ tmp = dbase[1:].split(';')
+ return PhysicalQuantity(str2number(tmp[0]), tmp[1])
+ else:
+ return str2number(dbase)
+ else:
+ raise ValueError("Broken FC unit in dbase: %s" % (dbase.__repr__(), ))
+
+def date2dbase(date):
+ """extends a short datestring to YYYY-MM-DD_hh:mm:ss.ssssss standard
+ """
+ assert len(date) in [4, 7, 10, 13, 16, 19, 21, 22, 23, 24, 25, 26]
+ date = date.replace(' ', '_')
+ complete_str = '0000-01-01_00:00:00.000000'
+ return date + complete_str[len(date):]
+
+def emd52type(emd5):
+ if emd5.endswith('d'):
+ return 'fc'
+ elif emd5.endswith('e'):
+ return 'sc'
+ else:
+ raise ValueError(emd5)
+
+def replace_type(str, type):
+ if type == 'field':
+ return str % ('fc', )
+ elif type == 'sample':
+ return str % ('sc', )
+
+def get_wildcards(length, char, braces=False, commas=True):
+ if braces:
+ wc = '('
+ else:
+ wc = ''
+ for index in xrange(length):
+ wc += char
+ if commas:
+ wc += ','
+ wc += ' '
+ if commas:
+ wc = wc[:-2]
+ else:
+ wc = wc[:-1]
+ if braces:
+ wc += ')'
+ return wc
+
+
+class AnyValue():
+ """Dummy class for use in search queries
+ """
+ def __init__(self):
+ pass
+
+
+class SQLiteWrapper(object):
+ """Wrapper class for DC meta data <-> sqlite3
+ """
+ common_keys = ['longname', 'shortname', 'machine',
+ 'creator', 'hash', 'date']
+ writable_keys = ['storage']
+ fast_keys = ['machine', 'creator', 'date', 'hash', 'type', 'id']
+ all_keys = ['id', 'hash', 'longname', 'shortname', 'machine', 'creator',
+ 'date', 'type', 'attributes', 'storage', 'unit', 'columns',
+ 'dimensions']
+ common_result_keys = common_keys + ['id', 'type', 'storage']
+ one_to_one_search_keys = ['longname', 'shortname', 'machine',
+ 'creator', 'hash', 'storage']
+ one_to_one_result_keys = one_to_one_search_keys + ['date', 'id', 'type']
+ common_search_keys = one_to_one_search_keys + ['id', 'attributes',
+ 'date_from', 'date_to']
+ fc_search_keys = common_search_keys + ['unit', 'dimensions']
+ sc_search_keys = common_search_keys + ['columns']
+ sortable_keys = common_keys + ['id', 'storage', 'type']
+ any_value = AnyValue()
+
+ def __init__(self, database, timeout=60.0):
+ """
+ Arguments:
+ - database: database to connect to
+ """
+ self.database = database
+ self.timeout = timeout
+ self.connection = None
+ self.cursor = None
+
+ def __enter__(self):
+ assert self.connection == None
+ assert self.cursor == None
+ self.connection = sqlite3.connect(self.database, self.timeout,
+ detect_types=sqlite3.PARSE_DECLTYPES)
+ self.cursor = self.connection.cursor()
+ return self
+
+ def __exit__(self, type, value, traceback):
+ if type == None:
+ try:
+ self.connection.commit()
+ except:
+ print "Could not commit changes to database."
+ if hasattr(self.cursor, 'close'):
+ self.cursor.close()
+ if hasattr(self.connection, 'close'):
+ self.connection.close()
+ self.cursor = None
+ self.connection = None
+
+ def __getitem__(self, emd5):
+ if self.has_entry(emd5):
+ if emd5.endswith('field'):
+ return FCRowWrapper(emd5, self.cursor)
+ elif emd5.endswith('sample'):
+ return SCRowWrapper(emd5, self.cursor)
+ raise KeyError(emd5)
+
+ def setup_dbase(self):
+ sqlite3.register_converter('QUANTITY', dbase2quantity)
+ def createTable(table_name, columns, cursor):
+ query = "CREATE TABLE IF NOT EXISTS %s (" % (table_name, )
+ for name, type in columns:
+ query += name + " " + type + ", "
+ query = query[:-2] + ")"
+ cursor.execute(query)
+ def createTrigger(trigger_name, action, table_name,
+ statements, cursor):
+ query = "CREATE TRIGGER IF NOT EXISTS %s AFTER %s ON %s "\
+ "FOR EACH ROW BEGIN %s END"
+ st_query = ''
+ for st in statements:
+ st_query += st + ';'
+ cursor.execute(query % (trigger_name, action,
+ table_name, st_query))
+ #create tables:
+ columns = [('sc_id', 'TEXT PRIMARY KEY UNIQUE NOT NULL'),
+ ('longname', 'TEXT'),
+ ('shortname', 'TEXT'),
+ ('machine', 'TEXT'),
+ ('creator', 'TEXT'),
+ ('date', 'TEXT'),
+ ('hash', 'TEXT'),
+ ('storage', 'TEXT')]
+ createTable("km_sc", columns, self.cursor)
+ columns[0] = ('fc_id', 'TEXT PRIMARY KEY UNIQUE NOT NULL')
+ columns.insert(7, ('unit', 'QUANTITY'))
+ columns.insert(8, ('bu_id', 'INT'))
+ createTable("km_fc", columns, self.cursor)
+ columns = [('sc_id', 'TEXT NOT NULL'),
+ ('fc_id', 'TEXT NOT NULL'),
+ ('fc_index', 'INT NOT NULL'),
+ ('', 'UNIQUE(sc_id, fc_id, fc_index)'),
+ ('', 'PRIMARY KEY(sc_id, fc_id, fc_index)')]
+ createTable("km_sc_columns", columns, self.cursor)
+ columns = [('fc_id', 'TEXT NOT NULL'),
+ ('dim_id', 'TEXT NOT NULL'),
+ ('dim_index', 'INT NOT NULL'),
+ ('', 'UNIQUE(fc_id, dim_id, dim_index)'),
+ ('', 'PRIMARY KEY(fc_id, dim_id, dim_index)')]
+ createTable("km_fc_dimensions", columns, self.cursor)
+ columns = [('dc_id', 'TEXT NOT NULL'),
+ ('key', 'TEXT NOT NULL'),
+ ('value', 'TEXT'),
+ ('', 'UNIQUE(dc_id, key)'),
+ ('', 'PRIMARY KEY(dc_id, key)')]
+ createTable('km_attributes', columns, self.cursor)
+ columns = [('dc_id', 'TEXT PRIMARY KEY UNIQUE NOT NULL')]
+ createTable("km_temporary", columns, self.cursor)
+ columns = [('bu_id', 'INTEGER PRIMARY KEY AUTOINCREMENT '\
+ 'NOT NULL UNIQUE'),
+ ('m', 'INT'),
+ ('g', 'INT'),
+ ('s', 'INT'),
+ ('A', 'INT'),
+ ('K', 'INT'),
+ ('mol', 'INT'),
+ ('cd', 'INT'),
+ ('rad', 'INT'),
+ ('sr', 'INT'),
+ ('EUR', 'INT'),
+ ('', 'UNIQUE(m, g, s, A, K, mol, cd, rad, sr, EUR)')]
+ createTable('km_base_units', columns, self.cursor)
+ #create triggers:
+ createTrigger('trigger_del_fc', 'DELETE', 'km_fc',
+ ['DELETE FROM km_attributes WHERE dc_id=OLD.fc_id',
+ 'DELETE FROM km_fc_dimensions WHERE fc_id=OLD.fc_id'],
+ self.cursor)
+ createTrigger('trigger_del_sc', 'DELETE', 'km_sc',
+ ['DELETE FROM km_attributes WHERE dc_id=OLD.sc_id',
+ 'DELETE FROM km_sc_columns WHERE sc_id=OLD.sc_id'],
+ self.cursor)
+ createTrigger('trigger_del_tmp', 'DELETE', 'km_temporary',
+ ['DELETE FROM km_fc WHERE fc_id=OLD.dc_id',
+ 'DELETE FROM km_sc WHERE sc_id=OLD.dc_id'],
+ self.cursor)
+ #clean tmp:
+ self.cursor.execute("DELETE FROM km_temporary")
+ #add IndexMarker dummy if not present yet:
+ from pyphant.core.DataContainer import IndexMarker
+ im = IndexMarker()
+ im_summary = {'id':'IndexMarker', 'longname':im.longname,
+ 'shortname':im.shortname, 'hash':im.hash,
+ 'creator':None, 'machine':None,
+ 'date':'xxxx-xx-xx_xx:xx:xx.xxxxxx',
+ 'unit':1, 'dimensions':['IndexMarker'], 'attributes':{}}
+ self.set_entry(im_summary, None)
+
+ def has_entry(self, id):
+ exe = self.cursor.execute
+ if id == 'IndexMarker':
+ type = 'fc'
+ else:
+ type = emd52type(id)
+ exe("SELECT %s_id FROM km_%s WHERE %s_id=?" % (type, type, type),
+ (id, ))
+ return self.cursor.fetchone() != None
+
+ def _set_fc_keys(self, insert_dict, summary):
+ exe = self.cursor.execute
+ insert_dict['fc_id'] = summary['id']
+ insert_dict['unit'] = quantity2dbase(summary['unit'])
+ try:
+ exe("INSERT OR ABORT INTO km_base_units "\
+ "(m, g, s, A, K, mol, cd, rad, sr, EUR) "\
+ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+ quantity2powers(summary['unit']))
+ l_row_id = self.cursor.lastrowid
+ except sqlite3.IntegrityError:
+ exe("SELECT bu_id FROM km_base_units WHERE m=? AND g=? "\
+ "AND s=? AND A=? AND K=? AND mol=? AND cd=? AND rad=? "\
+ "AND sr=? AND EUR=?", quantity2powers(summary['unit']))
+ tmp = self.cursor.fetchone()
+ assert tmp != None
+ l_row_id = tmp[0]
+ insert_dict['bu_id'] = l_row_id
+ dimension_query = "INSERT INTO km_fc_dimensions VALUES (?, ?, ?)"
+ for dim_id, dim_index in zip(summary['dimensions'],
+ range(len(summary['dimensions']))):
+ exe(dimension_query, (summary['id'], dim_id, dim_index))
+
+ def set_entry(self, summary, storage, temporary=False):
+ """Sets the meta data in the database according to the
+ summary dictionary. If the according entry already exists,
+ the database is not changed since the same emd5s should always
+ reference the same (meta)data.
+ Arguments:
+ - summary: dictionary with meta data
+ - storage: string type (e.g. path in local file system)
+ - temporary: Flag that marks data to be deleted upon next
+ call to setup_dbase().
+ """
+ if self.has_entry(summary['id']):
+ return
+ exe = self.cursor.execute
+ if temporary:
+ exe("INSERT INTO km_temporary VALUES (?)",
+ (summary['id'],))
+ insert_dict = dict([(key, value) for key, value in \
+ summary.iteritems() if key in \
+ SQLiteWrapper.common_keys])
+ insert_dict['storage'] = storage
+ insert_dict['date'] = date2dbase(insert_dict['date'])
+ if summary['id'] == 'IndexMarker':
+ type = 'fc'
+ else:
+ type = emd52type(summary['id'])
+ attr_query = "INSERT INTO km_attributes VALUES (?, ?, ?)"
+ for key, value in summary['attributes'].iteritems():
+ assert isinstance(key, StringTypes)
+ exe(attr_query, (summary['id'], key, value.__repr__()))
+ if type == 'fc':
+ self._set_fc_keys(insert_dict, summary)
+ else:
+ insert_dict['sc_id'] = summary['id']
+ column_query = "INSERT INTO km_sc_columns VALUES (?, ?, ?)"
+ for fc_id, fc_index in zip(summary['columns'],
+ range(len(summary['columns']))):
+ exe(column_query, (summary['id'], fc_id, fc_index))
+ insert_query = "INSERT INTO km_%s %s VALUES %s"
+ value_list = []
+ key_query = "("
+ value_query = "("
+ for key, value in insert_dict.iteritems():
+ value_query += "?, "
+ key_query += key + ", "
+ value_list.append(value)
+ key_query = key_query[:-2] + ")"
+ value_query = value_query[:-2] + ")"
+ insert_query = insert_query % (type, key_query, value_query)
+ exe(insert_query, tuple(value_list))
+
+ def get_emd5_list(self):
+ self.cursor.execute("SELECT fc_id FROM km_fc")
+ emd5_list = self.cursor.fetchall()
+ self.cursor.execute("SELECT sc_id FROM km_sc")
+ emd5_list.extend(self.cursor.fetchall())
+ return [row[0] for row in emd5_list]
+
+ def verify_keys(self, keys, allowed):
+ for key in keys:
+ if not key in allowed:
+ raise KeyError(key)
+
+ def translate_result_key(self, key, type):
+ if key == 'id':
+ return replace_type("%s_id", type)
+ elif key == 'type':
+ return "'%s' AS type" % type
+ else:
+ return key
+
+ def translate_unit_search(self, value):
+ if isinstance(value, PhysicalQuantity):
+ value = value.unit.powers
+ elif isinstance(value, (IntType, LongType, FloatType)):
+ value = [0] * 10
+ elif isinstance(value, PhysicalUnit):
+ value = value.powers
+ else:
+ raise ValueError(value)
+ expr = '(bu_id IN (SELECT bu_id FROM km_base_units WHERE '\
+ 'm=? AND g=? AND s=? AND A=? AND K=? AND mol=? '\
+ 'AND cd=? AND rad=? AND sr=? AND EUR=?))'
+ return (expr, value, True)
+
+ def translate_attr_search(self, value, type):
+ expr = '('
+ new_value = []
+ for attr_key, attr_value in value.iteritems():
+ if isinstance(attr_value, AnyValue):
+ value_expr = ''
+ new_value.append(attr_key)
+ else:
+ value_expr = ' AND value=?'
+ new_value.extend([attr_key, attr_value.__repr__()])
+ expr += '(%s IN (SELECT dc_id FROM km_attributes '\
+ 'WHERE key=?%s))' \
+ % (replace_type('%s_id', type), value_expr)
+ expr += ' AND '
+ expr = expr[:-5] + ')'
+ return (expr, new_value, True)
+
+ def translate_list_search(self, key, value, type):
+ id_str = replace_type('%s_id', type)
+ if key == 'columns':
+ table = 'km_sc_columns'
+ index_str = 'fc_index'
+ lid_str = 'fc_id'
+ else:
+ table = 'km_fc_dimensions'
+ index_str = 'dim_index'
+ lid_str = 'dim_id'
+ qry = '(%s IN (SELECT %s FROM %s WHERE %s=? AND (%s IN (%s))))'\
+ % (id_str, id_str, table, index_str, lid_str, '%s')
+ new_value = []
+ expr = '('
+ for fc_search_dict, fc_index in zip(value, range(len(value))):
+ fc_query, fc_values = self.get_andsearch_query(
+ 'field', ['id'], fc_search_dict, False)
+ expr += qry % (fc_query, )
+ new_value.append(fc_index)
+ if fc_values != None:
+ new_value.extend(fc_values)
+ expr += ' AND '
+ return (expr[:-5] + ')', new_value, True)
+
+ def translate_search_dict(self, type, search_dict):
+ where = ''
+ values = []
+ for key, value in search_dict.iteritems():
+ extend = False
+ if key in self.one_to_one_search_keys:
+ expr = '%s=?' % key
+ elif key == 'id':
+ expr = '%s=?' % replace_type('%s_id', type)
+ elif key == 'date_from':
+ expr = 'date>=?'
+ value = date2dbase(value)
+ elif key == 'date_to':
+ expr = 'date<?'
+ value = date2dbase(value)
+ elif key == 'unit':
+ expr, value, extend = self.translate_unit_search(value)
+ elif key == 'attributes':
+ expr, value, extend = self.translate_attr_search(value, type)
+ elif key == 'columns' or key == 'dimensions':
+ expr, value, extend = self.translate_list_search(
+ key, value, type)
+ else:
+ raise NotImplementedError(key)
+ where += expr + " AND "
+ if extend:
+ values.extend(value)
+ else:
+ values.append(value)
+ return where[:-5], values
+
+ def get_andsearch_query(self, type, result_keys, search_dict, distinct):
+ trans_res_keys = tuple([self.translate_result_key(key, type) \
+ for key in result_keys])
+ if type == 'field':
+ table = 'km_fc'
+ elif type == 'sample':
+ table = 'km_sc'
+ if search_dict == {}:
+ qry = "SELECT%s %s FROM %s "
+ values = None
+ else:
+ qry = "SELECT%s %s FROM %s WHERE "
+ if distinct:
+ dist_str = ' DISTINCT'
+ else:
+ dist_str = ' ALL'
+ qry = (qry % (dist_str, get_wildcards(len(trans_res_keys),
+ '%s'), table)) % trans_res_keys
+ if search_dict != {}:
+ where, values = self.translate_search_dict(type, search_dict)
+ qry += where
+ return qry, values
+
+ def get_andsearch_result(self, result_keys, search_dict={},
+ order_by=None, order_asc=True,
+ limit=-1, offset=0, distinct=False):
+ """returns a list of tuples filled with values of the result keys
+ matching the constraints of search_dict.
+ Arguments:
+ - result_keys: List (of length >= 1) of keys to include in the
+ result tuples.
+ - search_dict: Dict mapping keys to constraint values.
+ Use empty dict for no constraints at all
+ possible keys: values (used relational operator[, type constraint]):
+ 'longname': str types (==)
+ 'shortname': str types (==)
+ 'machine': str types (==)
+ 'creator: str types (==)
+ 'date_from:' str types:
+ YYYY[-MM[-DD[_hh:[mm:[ss[.s[s[s[s[s[s]]]]]]]]]]] (>=)
+ 'date_to:' str types:
+ YYYY[-MM[-DD[_hh:[mm:[ss[.s[s[s[s[s[s]]]]]]]]]]] (<)
+ 'hash': str types (==)
+ 'id': str types: emd5 (==)
+ 'type': 'field' or 'sample' (==)
+ 'attributes': dict mapping attr. key to attr. value (==)
+ use (SQLiteWrapper instance).any_value
+ or (KM instance).any_value to skip value check
+ 'storage': str types (==)
+ 'unit': PhysicalUnit or number or PhysicalQuantity (==, FC only)
+ 'dimensions': list of FC search dicts
+ (see above definitions, FC only)
+ 'columns': list of FC search dicts (see above definitions, SC only)
+ - order_by: element of result_keys to order the results by
+ or None for no special ordering
+ - order_asc: whether to order ascending
+ - limit: maximum number of results to return,
+ set to -1 for no limit, default: -1
+ - offset: number of search results to skip, default: 0
+ - distinct: flag that indicates whether the result list
+ should only contain distinct tuples.
+ Usage Examples:
+ Get list of all longnames:
+ get_andsearch_result(['longname'], distinct=True)
+ --> [('name1', ), ('name2', ), ...]
+ Get id and shortname of all FCs that are parametrized by
+ a time dimension along the primary axis:
+ tunit = PhysicalQuantity(1, 's')
+ get_andsearch_result(['id', 'shortname'],
+ {'type':'field',
+ 'dimensions':[{'unit':tunit}]})
+ --> [('emd5_1', 'name_1'), ('emd5_2', 'name_2'), ...]
+ """
+ if order_by == None:
+ order = ''
+ else:
+ assert order_by in result_keys
+ assert order_by in self.sortable_keys
+ order = ' ORDER BY %s' % order_by
+ if order_asc:
+ order += ' ASC'
+ else:
+ order += ' DESC'
+ assert isinstance(limit, int)
+ assert isinstance(offset, int)
+ if not search_dict.has_key('type'):
+ self.verify_keys(result_keys, self.common_result_keys)
+ self.verify_keys(search_dict.keys(), self.common_search_keys)
+ fc_query, fc_values \
+ = self.get_andsearch_query('field', result_keys,
+ search_dict, distinct)
+ sc_query, sc_values \
+ = self.get_andsearch_query('sample', result_keys,
+ search_dict, distinct)
+ if distinct:
+ dist_str = ''
+ else:
+ dist_str = ' ALL'
+ query = "%s UNION%s %s%s LIMIT %d OFFSET %d"
+ query = query % (fc_query, dist_str, sc_query, order, limit, offset)
+ if search_dict != {}:
+ values = fc_values + sc_values
+ else:
+ values = None
+ mod_search_dict = search_dict
+ else:
+ if search_dict['type'] == 'field':
+ allowed_search_keys = self.fc_search_keys
+ all...
[truncated message content] |
|
From: <zk...@us...> - 2009-09-28 10:34:07
|
Revision: 659
http://pyphant.svn.sourceforge.net/pyphant/?rev=659&view=rev
Author: zklaus
Date: 2009-09-28 10:33:55 +0000 (Mon, 28 Sep 2009)
Log Message:
-----------
Merge branch 'master' into svn-trunk
* master:
Bugfix: Account for relocated test file.
Modified Paths:
--------------
trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py
Modified: trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py
===================================================================
--- trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py 2009-09-28 10:18:07 UTC (rev 658)
+++ trunk/src/pyphant/pyphant/tests/TestKnowledgeManager.py 2009-09-28 10:33:55 UTC (rev 659)
@@ -72,10 +72,11 @@
os.remove(h5name)
def testGetHTTPFile(self):
- #host = "omnibus.uni-freiburg.de"
+ host = "omnibus.uni-freiburg.de"
+ remote_dir = "/~s8klzimm"
#remote_dir = "/~mr78/pyphant-test"
- host = "idefix.physik.uni-freiburg.de"
- remote_dir = "/~zklaus"
+ #host = "idefix.physik.uni-freiburg.de"
+ #remote_dir = "/~zklaus"
url = "http://" + host + remote_dir + "/knowledgemanager-http-test.h5"
# Get remote file and load DataContainer
filename, headers = urllib.urlretrieve(url)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
