The following problem was reported by David.Mansergh
@ Quantel.Com.
I have recently come across a problem writing AAF files.
This problem has occurred about 5 times in the last
month in various bits of our software that write AAF files.
Most recently it has occurred when using edlaaf. I am
hoping that this will allow the problem to be identified
and fixed!
test1.edl contains approximately 500 edits referring to
approximately 500 different tapes. edlaaf appears to
successfully convert this to test1.aaf (using the
command line "edlaaf 30 test1").
When test1.aaf is dumped using infodumper, the
following error is always
reported:
***Caught hresult 0x80030008*** ("There is insufficient
memory available to complete operation")
When test1.aaf is dumped using dump, the following
error is always
reported:
dump.exe: Error : "/Header-2/Content-3b03/Mobs-1901
{394}/Slots-4403{4}/" : There is insufficient memory
available to complete operation.
dump.exe: Fatal error in routine "dumpStorage".
IStorage::OpenStream() failed.
During the dumping process, Windows Task Manager
indicates that there is plenty of memory available.
I have created 10 AAF files from the same edl and only
about half of these exhibit this problem. Unfortunately, in
our other bits of software we do not have the luxury of
repeating the process until it works. I guess that the
unpredictability of the process is due to differences in
the aaf file (the mobIDs change if nothing else). I don't
know what determines the order of the mobs in the AAF
file but this appears to change between files.
I have tried this with various builds of the AAF SDK,
including aaf_release.Win.1_0_1.504.zip from
SourceForge.
Is anyone else aware of this problem and has it already
been fixed?
I would be grateful for any help that you might be able to
give. The attached zip contains test1.edl, test1.aaf,
edlaaf.log, infodumper.log, dump.log. Please say if you
need any more details.
Thanks in advance, David.
Logged In: YES
user_id=239292
The file(s) referred to in David Mansergh's comments were too
large to attach so I put them in the source forge AAF group
directory. scp them as follows:
scp
aaf.sourceforge.net:/home/groups/a/aa/aaf/htdocs/bugFiles/72
6916/files.zip <destination_dir>
This should work too:
http://aaf.sourceforge.net/bugFiles/726916/files.zip
Logged In: YES
user_id=239292
I was able to dump the file using the most recent version of
the SDK running on Linux. Both InfoDumper, axDump, and
DevUtils/dump worked,
InfoDumper and axDump both grew to ~75 MB in size - rather
large, considering they aren't doing very much. The dump
program only grew to ~2.1 Mb.
On my Win2K system, I see the same problem you
described when using InfoDumper and axDump. The
DevUtils/dump program, however, completed without
problem. InfoDumper and axDump both grew to about 64
MB before failing. The dump program grew to 12 MB.
Logged In: YES
user_id=239292
Phil Tudor's comments:
I have repeated your bug report when dumping using
Windows SS - it bombs out with insufficient memory at the
point you report. Note that, in this case, this file dumps *fine*
using the current Linux SS binary, which I believe is a
different piece of MS source code.
Logged In: YES
user_id=752351
> I have recently come across a problem writing AAF files.
Since the aaf file can be read correctly using the Linux SS
binary but not the Windows SS binary, it would appear that
the aaf file itself is fine. This would suggest that the bug is in
reading the aaf file, not writing it as I originally thought. I
suppose that it is comforting to know that the data has not
been lost - it's just not readily available!
Logged In: YES
user_id=567963
I can confirm what David has found. An application I wrote is
being used by a customer in Canada. He sent me an AAF
file which encountered this very same problem during an
import. It seems as though there is an unreasonably small
limit somewhere in the structured storage system.
The file I was sent contained over 13 thousand (yes,
THOUSAND) mobs - and that was only one half of the full
edited programme! The "insufficient memory" problem
cropped up at around 1600 mobs. In other words, only 20%
of the file could be processed before it fell over.
If there isn't an immediate fix available, does anyone know of
a workaround?
Logged In: YES
user_id=567963
In my earlier comment I remarked about a file which someone
sent me containing over 13,000 mobs. Needless to say, I
didn't count them. I was going by the InfoDumper printout.
However, I've just posted a bug re InfoDumper which is
causing totals to be displayed as hexadecimal figures when
in fact they're plain old decimal.
This means that the file actually contained about 3,000 mobs -
not 13,000 which in turn, means that roughly 50% of the
mobs were processed before the structured storage problem
caused everything to fall over. That's better I suppose but still
not really acceptable.
Logged In: YES
user_id=752351
At the AAF engineering meetings in New York, Tim Bingham
suggested that it would be worth checking if this problem
could be avoided by using &#8216;eager&#8217; rather than &#8216;lazy&#8217; loading.
This can be specified as a mode flag in
AAFFileOpenExistingRead(). I modified infodumper to specify
AAF_FILE_MODE_EAGER_LOADING but this caused the
following error: AAFRESULT_NOT_IN_CURRENT_VERSION.
The code in ImplAAFFile::OpenExistingRead() has the
following comment:
// Save the mode flags for now. They are not currently
(2/4/1999) used by the OM to open the doc file. Why do we
return an error if modeFlags != 0? Answer: because none of
them are implemented yet.
However, OMFile::openExistingRead()
allows &#8216;OMFile::eagerLoad&#8217; to be specified. When a new
version of AAFCOAPI.dll is built with this mode set in
ImplAAFFile::OpenExistingRead(), infodumper can
successfully dump the problematic files. Hurray!
Since this problem typically occurs in large AAF files, I
assume that eagerly loading these files is significantly less
than optimal. i.e. this is not the correct solution to the
problem. One small step for man, one giant leap for mankind?
Logged In: YES
user_id=788502
I found what on Win2k is impossible to create aaf files larger
2G - problem is in OMMSSStoredObject::createFile() and
OMMSSStoredObject::openFile() realisation. They use Win9x
compatible call to StgCreateDocfile() and StgOpenStorage().
After changing them to Win2K compatible calls
StgCreateStorageEx() and StgOpenStorageEx() with set up
sector size to 4096 the problem was fixed. But, in Standart
codecs source code (CAAFCDCICodec.cpp etc) you must
comment lines like
if ( offset + buflen > (aafPosition_t)
2*1024*1024*1024-1 ) {
return AAFRESULT_EOF;
}
I think - more right way to do this using autodetect of OS
(9x/Win2k/XP) and use appropriative calls.
VVS
Logged In: YES
user_id=239292
StgCreateStorageEx and StgOpenStorageEx are not
supported by the reference structured storage implementation
currently used on UNIX platforms.
The currently "sanctioned" method is to create AAF files that
reference essence data in excess of 2GB is to create a
sequence of source clip objects. Each source clip represents
a chunk of essence that is less than 2GB in size.
This is done using the IAAFMasterMob::CreateEssence(),
writing the essence data until the codec returns EOF, then
call IAAFMasterMobEx::ExtenEssence() and continue writing
the essence data. If the essence data is stored in external
files then this will work on all platforms, including those with
aging structured storage implementations, 2GB file system
limitations.
No extended interface is required to read the essence data.
The SDK recognizes a sequence of source clips and takes
appropriate actions to read the data.
This is demonstrated in the AAF/examples2/axMasterMobEx
example program.
Logged In: YES
user_id=788502
Thanks for you replay.
In my application I am already cut long clip on chunk less 2G.
But problem still exists if the essence is embedded - than the
total AAF try exceed 2G limit (2047Mb, not the essence - all
file) the exception is occured. Even in the case than I have a
number of short clips, summary length of which exceeded 2G.
When I create file using modified myself aafcoapi.dll, I can
create AAF any size and this file can be read by RC1 version
of aafcoapi.dll (another applications can read such files, back
compatibility exists).
I need to create aaf file for clips having duration more 10
minuts, much more.
One question more. Does in AAF file specification exists rule,
what one essence size must be less or equal 2G? I cant find,
but may be I am mistaken.
VVS
Logged In: YES
user_id=239292
There are, currently, practical concerns that limit the size of
a file to 2GB. It is not the result of any specification. The
problem is due to discord amoung the various structured
storage implementations, and the requirement to create AAF
files that can be read on all platforms.
For the moment, the solution is to store essence in external
files.
Logged In: YES
user_id=162067
Here is some information on this issue from Skip Pizzi at
Microsoft...
Skip Pizzi writes:
Here are some comments from Henry Lee at MS, which may
be of value to this discussion:
"I believe this a well-known limitation with structured
storage's shared memory heap used for custom marshaling,
which is set to 4MB per root open. For a single open file using
normal 512-byte sectors, there is limit of 4MB used to
allocate opened stream and storage objects. Typically, a
large number of concurrent open streams or storages will
trigger this problem.
Typical workarounds:
1. Revise the application to close it IStream or IStorage
objects immediately after using them. If using transacted
mode, try switching to direct mode. If there are stream or
storage pointer leaks, they need to be fixed. 2. Revise the
application to close and reopen its root storage object when
the memory error hits. 3. Switch to 4k-sector structured
storage files -- it doesn't have this limitation. 4. Use the
StgOpenStorageOnILockBytes or
StgCreateDocfileOnILockBytes APIs -- it doesn't have this
limitation. 5. Port the app to Win64 -- IA64 or AMD64
Windows doesn't have this limitation."
Also, if the reference to 'Linux SS binary' means the MS SS
Reference Implementation, it will behave differently, as Phil
reports. Henry Lee continues:
"The MS SS Reference Implementation shouldn't have this
problem, since it doesn't support marshaling and all allocations
are done using task memory.
The shared memory support is OLE (linking and embedding)
legacy. The 16-bit implementation used shared memory to
marshal storage and stream objects between processes very
efficiently, and this was carried over to 32-bit for
compatibility. On NT and later, the shared memory heap was
broken up into individual per-open shared memory heaps for
security and robustness reasons.
When the 4k-sector support was added in Win2k, this mode
used task memory instead of shared memory, so it could
handle a much greater number of concurrent objects in
memory. If AAF switches to the newer 4k-sector format, it
won't be subject to the docfile's internal shared-memory
limitations."