Thanks, Chris, I didn't see this until now.  Yeah, that appears to be the problem I'm having and the functionality that Asger is asking for would solve that problem.

I should be able to add the checksum to the datastream later, once that issue is resolved.

...adam
____________________________________________
Adam Wead
Systems and Digital Collections Librarian
Rock and Roll Hall of Fame and Museum
216.515.1960 (t)
215.515.1964 (f)


On Tue, Oct 25, 2011 at 4:55 PM, Chris Wilper <cwilper@duraspace.org> wrote:
Hi Adam,

I suspect you were running up against this problem, reported by Asger
a while ago:

https://jira.duraspace.org/browse/FCREPO-752

Your approach sounds pretty reasonable to me. Using "E" datastreams is
often done for repositories dealing with very large files for a
variety of reasons. And storing such datastreams in a BagIt structure
seems like a good idea on its own. Perhaps at some point when
FCREPO-752 is fixed, it will be more practical to *also* store fixity
info inside the Fedora objects.

I'm also interested in other people's opinions and experiences with
this kind of approach.

- Chris

On Tue, Oct 25, 2011 at 10:56 AM, Adam Wead <amsterdamos@gmail.com> wrote:
> Hi all,
>
> I'd like to throw out my latest strategy for storing external video files in
> Fedora and have you all pick at it, see if I'm doing something dumb.
>
> My first attempt used external datastreams (type E) with checksums.  This
> worked fine in test scenarios but I ran into problems with large video
> files.  I would get timeouts from the ruby code that I was using.  I'm
> developing a hydra head to ingest the video.  I was going to start digging
> in and figure out what was going on, but a weekend off made me think it
> might be better (and easier) to decouple these two things a bit more.
>
> The next idea I have is to use external datastreams but ditch Fedora's
> checksums.   I didn't have any problems when I tried this last week with
> large files, so I know it will work.  Obviously, I need the checksums
> somewhere, so I'm considering adopting a BagIt structure on the filesystem
> to store all the video files, or groups of video files, and then point
> fedora to the location of each file.
>
> Has anyone tried such a scenario with or without success?  I could always
> put a checksum file right alongside the video file and forgo the BagIt
> option, but I like the idea of having two independent but functional
> systems.  On one side Fedora which tracks the locations of the files and
> their metadata, and BagIt which tracks the binary data.
>
> As a matter of policy, I would use Fedora's checksums for data stored within
> Fedora, which we don't have at the moment, but I anticipate using for images
> and other smaller sized files.
>
> Any thoughts are appreciated.
>
> thanks,
>
> ...adam
>
> ____________________________________________
> Adam Wead
> Systems and Digital Collections Librarian
> Rock and Roll Hall of Fame and Museum
> 216.515.1960 (t)
> 215.515.1964 (f)
>
> ------------------------------------------------------------------------------
> The demand for IT networking professionals continues to grow, and the
> demand for specialized networking skills is growing even more rapidly.
> Take a complimentary Learning@Cisco Self-Assessment and learn
> about Cisco certifications, training, and career opportunities.
> http://p.sf.net/sfu/cisco-dev2dev
> _______________________________________________
> Fedora-commons-users mailing list
> Fedora-commons-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
>

------------------------------------------------------------------------------
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
Take a complimentary Learning@Cisco Self-Assessment and learn
about Cisco certifications, training, and career opportunities.
http://p.sf.net/sfu/cisco-dev2dev
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users