DSpace

Re: [Dspace-devel] Preservation data thoughts From: Jim Downing <ojd20@ca...> - 2004-08-18 16:01 Hi Rob, Strangely enough, I was just mulling some of these issues myself! On Wed, 2004-08-18 at 16:32, Tansley, Robert wrote: > As you may be aware, my main interest in DSpace is in using it for > digital preservation (see > http://wiki.dspace.org/index.php/HpPositionPaper). >=20 > I'm currently looking at the underlying data model DSpace uses, and > whether what we have in DSpace 1.2 is adequate. This throws up a > number of issues. >=20 > * Currently each bitstream can have one bitstream format. This > seems a little na=C3=AFve, since many individual bitstreams can > have a number of embedded components with different format.=20 > Some examples of this include: > * .zip, .tar.gz files. Just knowing you have a .zip > file won't let you use the contents of the file in the > future. > * .avi video files. The video and audio components > of files like this can have any one of a huge variety > of encodings (e.g. MPEG4 + AC3, Windows Media Video + > MP3). > * Office document files. E.g. Microsoft office > documents that have embedded OLE components, > powerpoint presentations with embedded video. > * HTML files with embedded Javascript, ActiveX controls, > and so forth. I think the best approach to this type of aggregate file will be to disaggregate as part of a canonicalization process. As you point out later, even defining the structure of such files will be a nightmare, let alone trying to write validators, migrators and so on for every possible permutation. > I seem to remember Stephen Abrams mentioning something about > how JHOVE had considered this scenario at the DSpace user > group meeting in March, but I haven't dug anything up out of > the JHOVE site. I suspect we'll have to wait for JHOVE to publish their findings - I haven't seen anything on aggregate file types in the source. > * In many cases, simply knowing the format or encoding is not > enough; additional, semantic information is required.=20 > Consider the contents of a database. You may store the data > in an XML format to improve 'preservability', but just knowing > that file is an XML file is not enough. You need to know what > all the columns and relations mean. Getting this sort of > information out of end users is not going to be easy. To an extent, and in certain circumstances, this could be solved by enforcing standards (e.g. enforcing a certain XML schema who's semantics are known). This would mean writing identification and validation routines that go beyond syntactic checking. Those are fairly heavy caveats, though, and not practical for legacy content. > * What does one do about 'broken' files? Take the HTML > standard. There are plenty of essentially proprietary > extensions to this, and plenty of pages which 'work' (perhaps > in a subset of available browsers) despite being 'broken' (not > following the standard in some regard). Should such 'broken' > bitstreams be flagged as such, so that in the future, > something reading it knows to expect some weirdness? Absolutely. This is especially relevant in situations where some entity has attempted to deliberately subvert / extend an existing standard in an attempt to own a new de-facto standard. The examples that spring to mind are Netscape and MS with HTML and MS with the Java Virtual Machine. Hmmm, there's a common thread there! ;-) > * So-called 'complex' or 'conglomerate' objects -- i.e. objects > containing several bitstreams. The DSpace 2.0 proposal calls > for a 'structure map' such as that offered by METS. However, > is one structure map model ever going to be enough?=20 > Requirements will vary from situation to situation and over > time? > * How on Earth are we going to get any of the required > information out of end users? > I'm wondering how (and whether) the DSpace data model needs to evolve > to support preservation (as well as the numerous other uses of > DSpace!) Currently the DSpace 2.0 design proposal suggests replacing > the DSpace 1.2 notion of bundles with a structure map, something like > is present in METS right now. However this may be only part of the > story -- do we need to rethink the way bitstreams and bitstream > formats are handled? I think that if you disaggregate complex objects then the structure map approach should cope. Bistream formats definitely need some work. I'm quite persuaded by the TOM approach, identifying a format by the operations that can be done on them. (My apologies if I've got that spectacularly wrong and John Ockerbloom is reading!). jim

Re: [Dspace-devel] Preservation data thoughts From: Jerome McDonough <jerome@ny...> - 2004-08-18 17:16 Hi Robert, The issue of embedded bitstreams that you raised has come up in a couple of different contexts for me recently. There is a working group, formed by OCLC and RLG, looking at preservation metadata for digital objects, and this issue of embedded bitstreams and how we record preservation metadata for them has been a focus of discussion for several weeks. It has also come up in some informal discussions among those of us in the digital library community working on digital video assets, for the reasons you note. These are complicated, nasty issues, and while we're asking the same questions you are, I can't say we really have any great answers yet. I'm *definitely* not the best person to talk about the DSpace data model, but speaking from the METS world, I can attest that the structural map was not intended to be a mechanism for associating technical or preservation metadata regarding an asset and/or its subcomponents with data files. All of that is handled by a separate (and admittedly, somewhat primitive) mechanism in METS, which allows multiple instances of administrative metadata to be associated with a single file; you could have a video file, for example, which had separate video technical metadata, audio technical metadata, and IP rights metadata associated with it. To some degree, the METS structural map was really intended as a mechanism to allow people to create complex, composite objects *without* having to drop all of the bitstreams in a single file. The underlying assumption was that for preservation, simpler is better, and that a file with a single 'essence' bitstream (and possibly associated embedded metadata) was going to be easier/cheaper to deal with in the long term than complex, compound objects in proprietary file formats. On the question of 'Is one structural map enough?' the METS community answered 'no,' which is why METS supports multiple structural maps for a single object. If SGML, with the ability to express concurrent, overlapping structural hierarchies on a single document, had won the day, a single structural map might be sufficient, but in the XML world, you really need to support multiple structural maps to accomplish that goal. We are in the very early stages of adopting DSpace at NYU, but we are planning on trying to contribute to the DSpace community by working on issues surrounding making DSpace "METS-friendly," including support for structural maps, more sophisticated handling of technical metadata, and possibly support for METS' "digital provenance" preservation metadata. We would love to talk with other people interested in these areas and learn what other people are doing, so that we can focus our own development efforts on areas that will be of widest use to the community. all the best, Jerome McDonough Digital Library Development Team Leader Elmer Bobst Library, New York University 70 Washington Square South New York, NY 10012 (212) 998-2425 jerome@...

1 message has been excluded from this view by a project administrator.