#329 Need special dispensation to upload 3 large packages

v1.0_(example)
closed
nobody
None
5
2014-12-11
2014-01-06
No

Astronomers using my jplephem package have previously been able to download the five data packages that can be used with it, de405, de406, de421, de422, and de423, with pip install that pulled from five .tar.gz URLs that I was hosting on my Amazon S3 account.

But pip is apparently now refusing to install from the external URLs, so I would like to simply do a one-time upload of these packages to PyPI. The two packages de421 and de423 have uploaded without problems, but the three packages de405, de406, and de422 gave me the following error:

Upload failed (400): distribution file too large

Could my brandonrhodes account be given special one-time dispensation to upload these three packages, despite their large size, so that these data sets are easy to install for astronomers who might be very new to Python and might not know how to navigate pip and its increasingly stringent restrictions about package install? Thanks!

Discussion

  • Richard Jones
    Richard Jones
    2014-01-06

    So those files are just data? How large are the files? We don't currently have a per-user exception capability for file size, so I'd have to add it. I wonder whether there could be a different approach (perhaps a download facility built into jplephem itself?)

     
  • Two of the files are already uploaded because they fell under the quota, in case you want to see a live example:

    https://pypi.python.org/pypi/de421
    https://pypi.python.org/pypi/de423

    They are, indeed, data: a small init.py combined with a series .npy files which contain NumPy arrays tightly serialized to disk in the native NumPy external storage format.

    The reason that I wanted these particular ephemerides on PyPI is so that astronomy projects could depend on them directly. I am happy to have all sorts of other data sets be downloaded through side channels, but data that is necessary to a program’s operation seemed like the kind of thing that should be listed in their install dependencies and placed on the system through the same mechanism that populates a virtualenv with all of a package’s other necessary apparatus.

    After all, adding a download mechanism to jplephem, along with a URL scheme and hosting and security and versioning and upgrading and integration with Python install tools, would essentially be re-inventing PyPI and pip all over again all by myself. So my hope was to go ahead and use the existing tools instead of trying to patch together my own. :)

     
  • Richard Jones
    Richard Jones
    2014-01-11

    Hm, OK. What is the actual file size of the files being rejected?

     
  • The three files being rejected are roughly:

    de405 — 54M
    de406 — 178M
    de422 — 544M

    Yes, the Jet Propulsion Laboratory went a little crazy with that last one there. They wanted to create a long-term reference ephemeris that would cover the planets’ motion from 3000 BC to 3000 AD.

     
  • Richard Jones
    Richard Jones
    2014-01-11

    OK, I'll put something in place in the next day that should allow you to host those large files.

     
    • "Richard Jones" richard@users.sf.net writes:

      OK, I'll put something in place in the next day that should allow you
      to host those large files.

      An attempt to upload still results in an error message, in case you have
      been waiting for me to test this out again:

      $ python setup.py sdist upload (master*)
      ~/python-jplephem/de405
      running sdist
      running check
      warning: sdist: manifest template 'MANIFEST.in' does not exist (using
      default file list)

      warning: sdist: standard file not found: should have one of README,
      README.txt

      writing manifest file 'MANIFEST'
      creating de405-1997.1
      creating de405-1997.1/de405
      making hard links in de405-1997.1...
      hard linking setup.py -> de405-1997.1
      hard linking ...
      Creating tar archive
      removing 'de405-1997.1' (and everything under it)
      running upload
      Submitting dist/de405-1997.1.tar.gz to http://pypi.python.org/pypi
      Upload failed (400): distribution file too large

      --
      Brandon Rhodes brandon@rhodesmill.org http://rhodesmill.org/brandon

       
  • Richard Jones
    Richard Jones
    2014-01-31

    Sorry Brandon, things came up. I'm looking at the code at the moment and I'm not sure I can actually support even hundreds of megabytes easily - I think the front-end web server will probably block the request as part of standard DOS protections. I'll check with the infrastructure people.

     
    • Well, wait - do we really need to change the code? I mean, if I were to
      "setup.py register" and then, you know, email you the new .tar.gz files
      or something, could they just be put in place in a few seconds without
      you having to specially make an exception to the upload process? :)

      --
      Brandon Rhodes brandon@rhodesmill.org http://rhodesmill.org/brandon

       
  • Richard Jones
    Richard Jones
    2014-01-31

    Hm, I guess if you could get the file to me somehow - emailing a 544MB file seems unlikely, but if you can host it somewhere for me to download that could work. I'd have to do additional hacks to register the file upload internally, but as long as the files don't change frequently...

     
    • Sorry for the delay on helping you close this issue; I was going to
      switch to more explicit external hosting for these files, but the pip
      user experience is now poor enough for externally-hosted files that
      these packages do need to be present on PyPI itself. Here are links to
      the .tar.gz files, if you could please manually add these to the two
      packages that they belong to - thanks! (And, again, these are packages
      which should never ever change, so you will not wind up having to host
      multiple versions of them on PyPI!)

      http://jplephem.s3.amazonaws.com/de406-1997.1.tar.gz
      http://jplephem.s3.amazonaws.com/de422-2009.1.tar.gz

      Thank you!

      --
      Brandon Rhodes brandon@rhodesmill.org http://rhodesmill.org/brandon

       
  • Richard Jones
    Richard Jones
    2014-06-16

    • status: open --> closed
     
  • Richard Jones
    Richard Jones
    2014-06-16

    This is now done.