#2105 support github with custom filesystem

[Jeremy Faden]

I keep forgetting that I can't run scripts directly from github, and having such an area would be great to support users with working documentation. The problem is the raw view which should work in Autoplot doesn't work because the parent doesn't exist.

[Jeremy Faden]

Maybe even a mirror, where we copy the github folders every minute. I'm trying to find a point digitizer, and this should be trivial for me.

[Jeremy Faden]

I have this working fairly well. One problem is that file dates are not returned in the metadata, so I've finally put in support for ETags as well.

[Jeremy Faden]

I've made it so that when a filesystem for https://github.com/autoplot/jyds/tree/master/annotations/ is created, the extra "tree/master" is removed. This will allow easier copy-n-paste from a browser.

[Jeremy Faden]

I verified that the "link to Local Read-Only Cache" can be used to link to a checked out version of the github directory, allowing a local experiments before committing new versions to github. It would be a fun project to visualize differences between the remote and local copies of the code.

[Jeremy Faden]

See also https://sourceforge.net/p/autoplot/feature-requests/665/

[Jeremy Faden]

vap files show up in a strange position. See https://github.com/autoplot/vap/blob/gh-pages/geos.vap

[Jeremy Faden]

I didn't realize how close Das2's http filesystem is to supporting files like
https://raw.githubusercontent.com/tsds/tsds2/gh-pages/tsdsfe/scripts/tsdsfe.jyds

The HttpFileSystem goes immediately into offline mode, but one can still use the FileSystem object to get the file because ETags are now cached. The following script is able to get the remote file and print its length:

# Bob and I were wondering why Autoplot can't get a file if it can't list it.  I need to
# review this.

from org.das2.util.filesystem import FileSystem
FileSystem.reset()

# In this step, Das2 attempts to connect to the root, so that authentication can be done.
fs= FileSystem.create('https://raw.githubusercontent.com/autoplot/scripts/master/')

fo= fs.getFileObject('makeTutorialHtml.jy')
f= fo.getFile(monitor)

print 'length=', f.length()

Something prevents this from use as a data source, and I'm looking into that now.

[Jeremy Faden]

Files can now be used with http without having to list the parent folder.

[Jeremy Faden]

Masafumi has a file from RadioJove which isn't loading, because its parent returns forbidden. I need to get this URI from him.

[Jeremy Faden]

The GitHubFileSystem works well, except that it depends on etags and I'm not sure that these work reliably in Java. I have a case where

https://github.com/autoplot/dev/blob/master/bugs/2156/demo2156.jyds

was not seeing updates I was making on the website.

[Jeremy Faden]

[Jeremy Faden]

I need to reopen this, because Java is doing some caching which is incorrect (I believe). If I edit a script directly on github, it takes about a minute for the changes to be respected by the client's Java.

[Jeremy Faden]

I was playing with: https://github.com//autoplot/dev/raw/master/rfe/sf/545/example1.jy

[Jeremy Faden]

See also https://sourceforge.net/p/autoplot/bugs/2203/, which will be the ticket for the latency problem.

[Jeremy Faden]

See also https://sourceforge.net/p/autoplot/feature-requests/713/, where support for branches would be added.

[Jeremy Faden]

At some point I started support for working on branches, and this broke support for where we would link to a local readonly copy, and this would allow the me or Masafumi to experiment with local changes before pushing them.

This is fixed, but the code assumes that the local copy is on the master branch. This might be ideal behavior, if one assumes that the goal is to make changes to the master branch.

[Jeremy Faden]

A lot of the above was done. For example, branches are supported when direct links are used. Also the strange delay to updates is a 300 second delay, and this is described in another ticket.