#528 Jython namespaces path names need to be changed, removing virbo name

[Jeremy Faden]

The package names need to be updated, but this is not trivial because of old jython scripts that refer to Java packages explicitly. This is a list of all the imports and a place to store the namespace changes.

I know this is a repeat ticket, but I can't find the old one.

[Jeremy Faden]

1. QStream and QDataSet projects will become part of das2 and will be housed at U. Iowa's server.
2. package names containing virbo will be changed.

Old Name	New Name	Notes
org.virbo.autoplot	org.autoplot
org.virbo.autoplot.state	org.autoplot.state
org.virbo.datasource	org.autoplot.datasource
org.virbo.dataset	org.das2.qds
org.autoplot.bufferdataset	org.das2.qds.buffer
org.qdataset	org.das2.qds
org.virbo.dsops	org.das2.qds.ops
org.virbo.dsutil	org.das2.qds.util
org.virbo.filters	org.das2.qds.filters
org.autoplot.pngwalk	org.autoplot.pngwalk	no need for change
org.virbo.qstream	org.das2.qstream
org.qstream.filter	org.das2.qstream.filter
org.virbo.ascii	org.autoplot.ascii
org.virbo.das2Stream	org.autoplot.das2stream	(note lower case)
org.virbo.spase	org.autoplot.spase
zipfs	org.das2.util.filesystem	NOT DONE
ftpfs	org.das2.util.filesystem	NOT DONE
org.virbo.autoplot from servlet	org.autoplot.servlet	assume no use in scripts so name change is safe.

TO BE CONSIDERED LATER:
* org.das2.qds will be the interface and a few implementations, sufficient to support das2 apps.
* org.das2.qdsops will be simple operators and depends only on org.das2.qds
* org.das2.qdsutil will be more abstract operations (load ascii file) and depends on the remainder of the package.
* note org.virbo.dataset contains DataSetOps and DataSetUtil, which should be moved appropriately.

[Jeremy Faden]

Search for imports in scripts:

from org.virbo.autoplot.AutoplotUtil import isParsableDouble
from org.virbo.autoplot.bookmarks import Bookmark
from org.virbo.autoplot.dom import ChangesSupport
from org.virbo.autoplot.dom import DomOps
from org.virbo.autoplot.dom import DomUtil
from org.virbo.autoplot.dom.Plot import PROP_CONTEXT
from org.virbo.autoplot import APSplash
from org.virbo.autoplot import AutoplotUI
from org.virbo.autoplot import AutoplotUtil
from org.virbo.autoplot import RenderType
from org.virbo.autoplot import ScreenshotsTool
from org.virbo.autoplot.layout import LayoutConstants
from org.virbo.autoplot.scriptconsole import DumpRteExceptionHandler
from org.virbo.autoplot.state import StatePersistence
 from org.virbo.autoplot.transferrable import ImageSelection
from org.virbo.dataset.DataSetOps import unbundle
from org.virbo.dataset.DataSetUtil import asDataSet, asArrayOfDoubles, as2DArrayOfDoubles
from org.virbo.dataset.DataSetUtil import gcd
from org.virbo.dataset.DataSetUtil import guessCadence
from org.virbo.dataset.DataSetUtil import guessCadenceNew
from org.virbo.dataset.DataSetUtil import kmeansCadence
from org.virbo.dataset.examples import Schemes
from org.virbo.dataset import ArrayDataSet
from org.virbo.dataset import BundleDataSet
from org.virbo.dataset import CdfSparseDataSet
from org.virbo.dataset import DataSetAnnotations
from org.virbo.dataset import DataSetOps
from org.virbo.dataset import DataSetOps
from org.virbo.dataset import DataSetUtil
from org.virbo.dataset import DataSetUtil
from org.virbo.dataset import DataSetUtil
from org.virbo.dataset import DataSetUtil
from org.virbo.dataset import DataSetUtil
from org.virbo.dataset import DDataSet
from org.virbo.dataset import DDataSet
from org.virbo.dataset import IDataSet
from org.virbo.dataset import LeafTrimDataSet
from org.virbo.dataset import NearestNeighborTcaFunction
from org.virbo.dataset import QubeDataSetIterator
from org.virbo.dataset import SemanticOps
from org.virbo.dataset import SemanticOps
from org.virbo.dataset import SparseDataSet
from org.virbo.dataset import SparseDataSet, SparseDataSetBuilder
from org.virbo.dataset import TrimDataSet
from org.virbo.dataset.SemanticOps import getUnits
from org.virbo.dataset.SemanticOps import lookupUnits
from org.virbo.datasource.capability import TimeSeriesBrowse
from org.virbo.datasource.DataSetURI import getHtmlFile
from  org.virbo.datasource import AutoplotSettings
from org.virbo.datasource import AutoplotSettings
   from org.virbo.datasource import DataSetURI
from org.virbo.datasource import DataSetURI
from org.virbo.datasource import DataSetURI as dsutil
from org.virbo.datasource import DataSourceRegistry
from org.virbo.datasource import DataSourceRegistry 
from org.virbo.datasource import DataSourceRegistry,DataSourceEditorPanelUtil
from org.virbo.datasource import DataSourceUtil
from org.virbo.datasource import FileSystemUtil
from org.virbo.datasource import ReferenceCache
   from org.virbo.datasource import URISplit
from org.virbo.datasource import URISplit
from org.virbo.demos import PlasmaModel
from org.virbo.dsops import CoerceUtil
from org.virbo.dsops.MyOps import rotate
from org.virbo.dsops.Ops import FFTFilterType
from org.virbo.dsops.Ops import safeName
from org.virbo.dsutil import AsciiParser
from org.virbo.dsutil import AsciiParser,MultiFieldTimeParser
from org.virbo.dsutil import AsciiParser, MultiFieldTimeParser
from org.virbo.dsutil import BundleBuilder
from org.virbo.dsutil import DataSetBuilder
from org.virbo.dsutil import DataSetBuilder
from org.virbo.dsutil import LinFit
from org.virbo.dsutil import LSpec
#from org.virbo.dsutil import Reduction
from org.virbo.dsutil import Reduction
from org.virbo.idlsupport import APDataSet
from org.virbo.imagedatasource import ImageDataSet
from org.virbo.imagedatasource import Util  
from org.virbo.jythonsupport import JythonUtil
from org.virbo.jythonsupport.JythonUtil import simplifyScriptToGetParams
#import org.virbo.dataset.ArrayDataSet.append as apd
import org.virbo.jythonsupport.ui.Util.FormData

[Jeremy Faden]

Note all the discovery mechanisms will need to be updated as well:

urls = ClassLoader.getSystemResources("META-INF/org.virbo.datasource.DataSourceFactory.extensions");

[Jeremy Faden]

My plan is to first implement the reverse map, so new names will be translated to old names, so I can discover any issues before breaking other peoples' codes. This could then be a last release of the 2016a branch, which would give some future support for the old branch with new scripts.

Also, some codes need to be imported that haven't been imported automatically. For example, SemanticOps should probably just be imported.

So here's a proposal for a Autoplot 2017 script:

from org.das2.qds.examples import Schemes
xx= xtags(Schemes.scalarTimeSeries())
from org.das2.qds.array import DDataSet
ds= DDataSet.createRank1(xx.length())
ds= putProperty( ds, QDataSet.DEPEND_0, xx )

2017-06-27: Note that DDataSet was left in org.das2.qds, otherwise refactoring would be more difficult.

[Jeremy Faden]

See also
https://sourceforge.net/p/autoplot/feature-requests/474
which is the old ticket.

[Jeremy Faden]

.vap files contain references to paths that will need to be corrected.

[Jeremy Faden]

I think org.das2.qds.array is too difficult, because there might be

from org.virbo.dataset import IndexGenDataSet,DDataSet

So we should have a one-to-one-and-onto (bijective) mapping.

[Jeremy Faden]

This refactoring should also take care of the BoxSelectionListener class BoxSelected method, which I've never been able to fix because of legacy scripts.

For example:

def boxSelected(event):
       ev1= createEvent( event.getXRange().toString(),0xFF0000, 'Turn on')
       dpr.addDataPoint( ev1 )

mm.BoxSelected=boxSelected

from nudnik.physics.uiowa.edu:/home/jbf/project/autoplot/script/lint/plotSymbolRedefined.jy

[Jeremy Faden]

IDL/Matlab scripts have references to full paths as well.

 apds= OBJ_NEW('IDLjavaObject$APDataSet','org.virbo.idlsupport.APDataSet')

[Jeremy Faden]

reverse map has the problem that working names (org.das2.util.filesystem.FileSystem) are converted to non-working names (zipfs.FileSystem). Any place where a bad name would be merged into a good name won't work, so the whole reverse map for v2016a will not work.

[Jeremy Faden]

Launchers like org.virbo.autoplot.AutoplotDataServer can be supported by leaving this package name and adapters that launch into the new names. (org.autoplot.pngwalk.PngWalkTool1 is an example that shows this is manageable.)

[Jeremy Faden]

I plan to commence this today. I will:

1. make a devel release 20170221a
2. make an svn tagged version of Autoplot v2016
3. open up everything in Netbeans.
4. change all package names, keeping a list as they are refactored (not the Eclipse list, however, just a text file).
5. update META-INF entries, discovery and launcher.
6. commit and run all tests.
7. add adapters for "main" launchers, including org.virbo.autoplot.(AutoplotUI,AutoplotServer,AutoplotDataServer)
8. add adapter for IDLMatlab bridge
9. enable the name map to support old Jython scripts.
10. Dan's platypus mac launcher will need to be fixed.

These will be the headers of the document at sftp://jfaden.net/home/jbf/ct/autoplot/rfe/528/worksheet.odt

[Jeremy Faden]

See also http://autoplot.org//autoplot2017

[Jeremy Faden]

Note, importing org.das2.util.filesystem.FileSystem has the effect of importing org.
sftp://jfaden.net/home/jbf/ct/hudson/script/test037/3577243.jy shows this problem, where there is no import of org.virbo.autoplot.RenderType to correct. I don't think this should be (or can be) corrected automatically. The script should be corrected manually to explicitly import the symbol.

AP> import org.das2.util.filesystem.FileSystem
AP> org.virbo.autoplot.RenderType
org.virbo.autoplot.RenderType

[Jeremy Faden]

I will not plan on refactoring inputs for the Autoplot "server" which is used to provide plotting to IDL and Matlab. I don't think this gets a lot of use, and I don't think people really need to import things (though this is allowed).

[Jeremy Faden]

The screenshots tool script showed where reflection was used (unnecessarly) to get a class reference. (I didn't know that org.autoplot.ScreenshotsTool was the class reference.) This will be another type of script that will not be corrected automatically.

from java.lang import Class
clas= Class.forName('org.virbo.autoplot.ScreenshotsTool')
from java.util.prefs import Preferences
prefs= Preferences.userNodeForPackage( clas )

should be simply:

from java.util.prefs import Preferences
prefs= Preferences.userNodeForPackage( ScreenshotsTool )

[Jeremy Faden]

Also with Java preferences, prefs are based on class path, so on Linux the preferences for org.virbo.autoplot.AutoplotUI are in ~/.java/.userPrefs/org/virbo/autoplot. These will not be supported, and I've been meaning to put these under the ~/autoplot_data directory anyway.

[Jeremy Faden]

Dan also showed that I can get at the old user preferences by using something like: Preferences.userRoot().node("/org/virbo/autoplot/")

I thought I was going to have to drop all the user default settings.

[Jeremy Faden]

Bob's jnlp script on autoplot.org will need to be modified.

[Jeremy Faden]

I've added a method AutoplotSettings.settings().getPreferences(Class) which will grab the old preferences. This will also make it easy to migrate to using ~/autoplot_data/config to store the preferences.

[Jeremy Faden]

I get this strange feedback when entering the from...import... statement:

fixImports in 0ms, affected=true.  Code contains imports with old ("virbo") names.
AP> from org.autoplot import AutoplotUI

Since I haven't referred to virbo, I would expect that I wouldn't see this message.

[Jeremy Faden]

In the script panel type in "from org.autoplot import <c>" and see that there's a runtime error:</c>

== Executing Script ==
Traceback (innermost last):
  (no code object) at line 0
  File "<string>", line 1
    from org.autoplot import 
                             ^
SyntaxError: invalid syntax
Traceback (innermost last):
  (no code object) at line 0
SyntaxError: ('invalid syntax', ('<string>', 1, 26, 'from org.autoplot import '))

error: Traceback (innermost last):
  (no code object) at line 0
  File "<string>", line 1
    from org.autoplot import 
                             ^
SyntaxError: invalid syntax

[Jeremy Faden]

Verify that the filters GUI works properly. It was left grey (pending)...

[Jeremy Faden]

A warning was issued incorrectly for any import of the form "from org.autoplot import AutoplotUI"