Re: [sword-app-tech] Asynchronous Deposit

[Richard J. <ri...@co...>]

Hi Sebastian,

One thing you could do in this situation is send back a deposit receipt as
soon as you can with a 202 (Accepted) header.  This at least means you can
continue to process the deposit asynchronously after the client has
finished sending data.  You may need to prepare all the relevant URIs, like
the Edit-IRI, and ensure that they return some suitable responses while the
item is still processing.

Technically, this status code is outside of the bounds of the spec, but we
did recommend a while back to introduce this as a feature and I hope that a
revision of the specification will be happening later this year where we
can make that change.

The other model I've come across is in use by DANS, which uses the
In-Progress header to allow you to send chunks of a full zip file in
successive, smaller, requests.  You can see their documentation for that
process here:

https://github.com/DANS-KNAW/easy-sword2#continued-deposit

The other thing I'd note is that multipart deposit hasn't turned out to
work very well for a lot of servers, so I'd avoid that mechanism if at all
possible.  Best approach, if you have an entry document, would be to send
the create with the entry document first, and then add the package later
via the edit media link.

Hope that helps.

Cheers,

Richard




On 21 June 2017 at 13:21, Sebastian Hofmann <hof...@un...> wrote:

> Hi,
>
> i used the JavaServer2.0 library to add Sword support to MyCoRe (
> http://mycore.de/) . We need Sword to transfer Files from Goobi (
> https://www.intranda.com/digiverso/goobi/) to MyCoRe based repositories.
> The implementation already works for small deposits, but we found some
> problems when we want to transfer bigger .zip Files up to 100gb+. We
> currently use the Multipart Deposit mechanism described in
> http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html.
> Our repository now needs much time to process these large deposits and
> send a deposit receipt back to the Client. For a 70gb .zip filled with
> uncompressed tif files our test repository needs 5 hours to validate,
> process and send back the deposit receipt. The connection is now 5 hours
> open without sending any data.
>
> Is there a best practice for sending such large deposits ? Should the
> client better send the .tiff sequential per .zip, or is there a way to send
> back a incomplete receipt with a "in proccessing" status ?
>
> Any suggestions would be great.
>
> Thanks
> Sebastian
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> sword-app-tech mailing list
> swo...@li...
> https://lists.sourceforge.net/lists/listinfo/sword-app-tech
>
>


-- 
Richard Jones,
Founder, Cottage Labs
https://cottagelabs.com || @cottagelabs

Lantern: https://lantern.cottagelabs.com
Repository Solutions: https://cottagelabs.com/repository