The VXL Project

Rob Radtke wrote:
> Hi everyone,
> I've been using VIL for a while now and had a question/comment for you 
> all. 
>  
> Unless I'm missing something, it appears as if VIL is unable to handle 
> image files larger than 2^31-1 bytes (i.e.. the largest number that can 
> be represented by vil_streampos which is typedef'd to be a long 
> int). The reason I noticed this in the first place was because I ran 
> across a NITF file that exceeded these bounds.  The NITF spec allows for 
> files as big as 10^12-1 (999,999,999,999) bytes. 
>  
> So, it seems as if this limitation is caused by the combination of two 
> things:
>  
> 1) vil_streampos is defined to be long int -- long int is 32 bits on my 
> platform (Windows).
> and
> 2) vil_stream has no API for iterative seeking (i.e.. seeking from the 
> current position as opposed to the start of the stream).
>  
> It seems like we could overcome this limitation by remedying either #1 
> or #2.  I took a crack at #1 in my working directory by changing the 
> typedef (of vil_streampos) to this:
>  
> #if VXL_HAS_INT_64
> typedef vxl_int_64 vil_streampos;
> #else //VXL_HAS_INT_64
> typedef long int vil_streampos;
> #endif //VXL_HAS_INT_64
>  
> That caused lots of compiler warnings elsewhere in VIL code base where 
> int, long int and vil_streampos are occasionally used interchangeably.  
> Some of these issues were easy to resolve and other would (I believe) 
> require API changes.  Also, vil_stream_fstream::seek() (which uses 
> std::fstream under the hood) would have to be modified.  My plan for 
> that was to have it do multiple seeks (from current position) for cases 
> where the desired seek offset exceeded the size of istream::streampos.  
> I never got this far.
>  
> The other option would be to attack #2 by adding a parameter similar to 
> ios_base::seekdir to vil_stream::seek().  Then client code could get a 
> larger seek, by iteratively seeking.  This would solve the NITF case 
> somewhat nicely (files up to 10^12 bytes) but would be infeasible for 
> REALLY large files where you may have to seek a ridiculous number of 
> times (worst case: 2^32) to get to the end of the file.  Of course, who 
> cares about files that are that big if they even exist.  This solution 
> requires a (small) change to the core API.
>  
> I suppose a third option would be to add a virtual seek64() function 
> that did the right thing for all subclasses.  I haven't really thought 
> this idea through too thoroughly.
>  
> At any rate, does anyone have any opinions on this?  Is there another 
> (easier) option I'm missing.  If I were to pursue a solution, would you 
> want it incorporated into VIL?  Thank you all for your consideration.
>  
> Rob Radtke

I think 1 sounds like the least hassle, and the best long term solution. 
However, the code currently assumes that vil_streampos has the same type 
as vcl_ios::streamoff. If it is generally the case that platforms with 
64 bit will define vcl_ios::streamoff as 64 bit, then there shouldn't be 
a problem. This is true for MSVC 7.1, and (according to gcc-bugzilla) 
for gcc 3.4.

Alternatively, just to be careful, vil_stream could #include<vcl_ios.h>
and typedef vcl_ios::streamoff vil_streampos.

I think the API changes you describe are worth doing - just convert all 
the ints that give errors/warnings to vil_streampos, and they should go 
away. This should include the calls to vil_stream_fstream::seek.

Now, the next question is, are you okay with only reading v.large NITF 
files on platforms with 64 bits?

Anyway, I'd be happy to accept such a patch with those fixes.

Ian.

Hi Ian,
Thank you for getting back with me.  I'm glad to hear that you're interested
in getting this resolved.  I'll send the patch to you when I've finished.
One major question though:

You had mentioned that MSVC 7.1 uses a 64 bit streamoff, but my research
seems to contradict this.  My Visual Studio documentation said only that:
"[streamoff's] representation has at least 32 value bits."  That wasn't very
definitive, so I checked the return value of 'std::numeric_limits<
vcl_ios::streamoff >::max()' and it was 2,147,483,647 (2^31 - 1).

Now, this isn't to say that solution #1 can't work.  I would just also have
to modify vil_stream_fstream::seek() to do multiple seeks when the desired
position exceeds 'std::numeric_limits< vcl_ios::streamoff >::max()'.  The
only problem I see with this is that the function could end up executing as
many as ~2^32 seeks (for the worst case of passing in a desired position of
2^63) -- that might be prohibitively slow.  I'm not sure where someone would
get access to a stream that big though (especially on a 32 bit system), so
it should be ok.  I.e. this approach would work nicely for all reasonable
cases.  What do you think?

Rob

->-----Original Message-----
->From: Ian Scott [mailto:ian.m.scott@...] 
->Sent: Friday, June 24, 2005 4:55 AM
->To: Rob Radtke
->Cc: vxl-maintainers@...
->Subject: Re: [Vxl-maintainers] VIL and large files
->
->Rob Radtke wrote:
->> Hi everyone,
->> I've been using VIL for a while now and had a 
->question/comment for you 
->> all. 
->>  
->> Unless I'm missing something, it appears as if VIL is 
->unable to handle 
->> image files larger than 2^31-1 bytes (i.e.. the largest 
->number that can 
->> be represented by vil_streampos which is typedef'd to be a long 
->> int). The reason I noticed this in the first place was 
->because I ran 
->> across a NITF file that exceeded these bounds.  The NITF 
->spec allows for 
->> files as big as 10^12-1 (999,999,999,999) bytes. 
->>  
->> So, it seems as if this limitation is caused by the 
->combination of two 
->> things:
->>  
->> 1) vil_streampos is defined to be long int -- long int is 
->32 bits on my 
->> platform (Windows).
->> and
->> 2) vil_stream has no API for iterative seeking (i.e.. 
->seeking from the 
->> current position as opposed to the start of the stream).
->>  
->> It seems like we could overcome this limitation by 
->remedying either #1 
->> or #2.  I took a crack at #1 in my working directory by 
->changing the 
->> typedef (of vil_streampos) to this:
->>  
->> #if VXL_HAS_INT_64
->> typedef vxl_int_64 vil_streampos;
->> #else //VXL_HAS_INT_64
->> typedef long int vil_streampos;
->> #endif //VXL_HAS_INT_64
->>  
->> That caused lots of compiler warnings elsewhere in VIL code 
->base where 
->> int, long int and vil_streampos are occasionally used 
->interchangeably.  
->> Some of these issues were easy to resolve and other would 
->(I believe) 
->> require API changes.  Also, vil_stream_fstream::seek() (which uses 
->> std::fstream under the hood) would have to be modified.  My 
->plan for 
->> that was to have it do multiple seeks (from current 
->position) for cases 
->> where the desired seek offset exceeded the size of 
->istream::streampos.  
->> I never got this far.
->>  
->> The other option would be to attack #2 by adding a 
->parameter similar to 
->> ios_base::seekdir to vil_stream::seek().  Then client code 
->could get a 
->> larger seek, by iteratively seeking.  This would solve the 
->NITF case 
->> somewhat nicely (files up to 10^12 bytes) but would be 
->infeasible for 
->> REALLY large files where you may have to seek a ridiculous 
->number of 
->> times (worst case: 2^32) to get to the end of the file.  Of 
->course, who 
->> cares about files that are that big if they even exist.  
->This solution 
->> requires a (small) change to the core API.
->>  
->> I suppose a third option would be to add a virtual seek64() 
->function 
->> that did the right thing for all subclasses.  I haven't 
->really thought 
->> this idea through too thoroughly.
->>  
->> At any rate, does anyone have any opinions on this?  Is 
->there another 
->> (easier) option I'm missing.  If I were to pursue a 
->solution, would you 
->> want it incorporated into VIL?  Thank you all for your 
->consideration.
->>  
->> Rob Radtke
->
->I think 1 sounds like the least hassle, and the best long 
->term solution. 
->However, the code currently assumes that vil_streampos has 
->the same type 
->as vcl_ios::streamoff. If it is generally the case that 
->platforms with 
->64 bit will define vcl_ios::streamoff as 64 bit, then there 
->shouldn't be 
->a problem. This is true for MSVC 7.1, and (according to gcc-bugzilla) 
->for gcc 3.4.
->
->Alternatively, just to be careful, vil_stream could 
->#include<vcl_ios.h>
->and typedef vcl_ios::streamoff vil_streampos.
->
->I think the API changes you describe are worth doing - just 
->convert all 
->the ints that give errors/warnings to vil_streampos, and they 
->should go 
->away. This should include the calls to vil_stream_fstream::seek.
->
->Now, the next question is, are you okay with only reading 
->v.large NITF 
->files on platforms with 64 bits?
->
->Anyway, I'd be happy to accept such a patch with those fixes.
->
->Ian.
->
->

Rob Radtke wrote:
> Hi Ian,
> Thank you for getting back with me.  I'm glad to hear that you're interested
> in getting this resolved.  I'll send the patch to you when I've finished.
> One major question though:
> 
> You had mentioned that MSVC 7.1 uses a 64 bit streamoff, but my research
> seems to contradict this.  My Visual Studio documentation said only that:
> "[streamoff's] representation has at least 32 value bits."  That wasn't very
> definitive, so I checked the return value of 'std::numeric_limits<
> vcl_ios::streamoff >::max()' and it was 2,147,483,647 (2^31 - 1).

My mistake, I just looked at the include files. It looked to me that if 
_WIN64 was defined, streamoff would be 64 bits, otherwise 32 bits. I 
guess that means that the processor/compiler may have 64 bits, but the 
OS still uses a 32 bit API. When a proper 64 bit Windows API is un use, 
then streamoff will go to 64 bits.

It strikes me that this is a limitation of the C++ standard. I guess 
strictly we need to add a VCL_SIZEOF_STREAMOFF into the config.

> Now, this isn't to say that solution #1 can't work.  I would just also have
> to modify vil_stream_fstream::seek() to do multiple seeks when the desired
> position exceeds 'std::numeric_limits< vcl_ios::streamoff >::max()'.  The
> only problem I see with this is that the function could end up executing as
> many as ~2^32 seeks (for the worst case of passing in a desired position of
> 2^63) -- that might be prohibitively slow.  I'm not sure where someone would
> get access to a stream that big though (especially on a 32 bit system), so
> it should be ok.  I.e. this approach would work nicely for all reasonable
> cases.  What do you think?

This would work. I think your worst case scenario is somewhat unlikely. 
Anyway if you jsut docuement the problem in the code, and tell users 
that they need to get a C++ implementation that has a 64 bit streamoff.

I'll be happy with your patch if the tests continue to work. I (and I 
guess most users) don't have any code that uses a vil_streampos directly 
- It's all hidden in the vil file loaders.

Ian.

Hi again,
So, I've come across an issue that I don't exactly know how to deal with.
We were discussing the necessary changes to vil_stream_fstream to get this
64 bit support working.  I still think those ideas have merit, but I haven't
figured out how I should implement vil_stream_fstream::file_size().  The
current implementation relies on vcl_fstream::tellg() like so:

vil_streampos vil_stream_fstream::file_size() const
{
  // if not already computed, do so
  if ( end_ == -1 ) {
    vcl_streampos curr = f_.tellg();
    f_.seekg( 0, vcl_ios_end );
    end_ = f_.tellg();
    f_.seekg( curr );
  }

  return end_;
} 

tellg() will be unable to return an accurate value for files larger than
2^31 bytes.  Does anyone know another, non-platform specific, way to
determine the size of a file on disk?  

Or, could we live with this function sometimes returning an inaccurate
value.  Typically, reading a large file (like a large NITF file) doesn't
require knowing the file size.  After all, you can't read the whole thing
into memory anyway.  Thoughts or suggestions?

Rob

->-----Original Message-----
->From: Ian Scott [mailto:ian.m.scott@...] 
->Sent: Friday, June 24, 2005 11:01 AM
->To: Rob Radtke
->Cc: Vxl-maintainers
->Subject: Re: [Vxl-maintainers] VIL and large files
->
->Rob Radtke wrote:
->> Hi Ian,
->> Thank you for getting back with me.  I'm glad to hear that 
->you're interested
->> in getting this resolved.  I'll send the patch to you when 
->I've finished.
->> One major question though:
->> 
->> You had mentioned that MSVC 7.1 uses a 64 bit streamoff, 
->but my research
->> seems to contradict this.  My Visual Studio documentation 
->said only that:
->> "[streamoff's] representation has at least 32 value bits."  
->That wasn't very
->> definitive, so I checked the return value of 'std::numeric_limits<
->> vcl_ios::streamoff >::max()' and it was 2,147,483,647 (2^31 - 1).
->
->My mistake, I just looked at the include files. It looked to 
->me that if 
->_WIN64 was defined, streamoff would be 64 bits, otherwise 32 bits. I 
->guess that means that the processor/compiler may have 64 
->bits, but the 
->OS still uses a 32 bit API. When a proper 64 bit Windows API 
->is un use, 
->then streamoff will go to 64 bits.
->
->It strikes me that this is a limitation of the C++ standard. I guess 
->strictly we need to add a VCL_SIZEOF_STREAMOFF into the config.
->
->> Now, this isn't to say that solution #1 can't work.  I 
->would just also have
->> to modify vil_stream_fstream::seek() to do multiple seeks 
->when the desired
->> position exceeds 'std::numeric_limits< vcl_ios::streamoff 
->>::max()'.  The
->> only problem I see with this is that the function could end 
->up executing as
->> many as ~2^32 seeks (for the worst case of passing in a 
->desired position of
->> 2^63) -- that might be prohibitively slow.  I'm not sure 
->where someone would
->> get access to a stream that big though (especially on a 32 
->bit system), so
->> it should be ok.  I.e. this approach would work nicely for 
->all reasonable
->> cases.  What do you think?
->
->This would work. I think your worst case scenario is somewhat 
->unlikely. 
->Anyway if you jsut docuement the problem in the code, and tell users 
->that they need to get a C++ implementation that has a 64 bit 
->streamoff.
->
->I'll be happy with your patch if the tests continue to work. I (and I 
->guess most users) don't have any code that uses a 
->vil_streampos directly 
->- It's all hidden in the vil file loaders.
->
->Ian.
->
->

Rob Radtke wrote:
> Hi again,
> So, I've come across an issue that I don't exactly know how to deal with.
> We were discussing the necessary changes to vil_stream_fstream to get this
> 64 bit support working.  I still think those ideas have merit, but I haven't
> figured out how I should implement vil_stream_fstream::file_size().  The
> current implementation relies on vcl_fstream::tellg() like so:
> 
> vil_streampos vil_stream_fstream::file_size() const
> {
>   // if not already computed, do so
>   if ( end_ == -1 ) {
>     vcl_streampos curr = f_.tellg();
>     f_.seekg( 0, vcl_ios_end );
>     end_ = f_.tellg();
>     f_.seekg( curr );
>   }
> 
>   return end_;
> } 
> 
> tellg() will be unable to return an accurate value for files larger than
> 2^31 bytes.  Does anyone know another, non-platform specific, way to
> determine the size of a file on disk?  
> 
> Or, could we live with this function sometimes returning an inaccurate
> value.  Typically, reading a large file (like a large NITF file) doesn't
> require knowing the file size.  After all, you can't read the whole thing
> into memory anyway.  Thoughts or suggestions?

Rob.

I won't be able to look into this properly for at least a week. Off 
the top of my head, I'd check if anything in vil (or anywhere else in 
VXL) uses file_size() - there's a good chance it doesn't. In that case 
just remove the file_size() API.

Alternatively, if the size can't be returned just print a helpful 
error message and abort. In the medium term, I'm tempted to add 
optional exceptions to vil to handle problems like this.

Ian.

> 
> Rob
> 
> ->-----Original Message-----
> ->From: Ian Scott [mailto:ian.m.scott@...] 
> ->Sent: Friday, June 24, 2005 11:01 AM
> ->To: Rob Radtke
> ->Cc: Vxl-maintainers
> ->Subject: Re: [Vxl-maintainers] VIL and large files
> ->
> ->Rob Radtke wrote:
> ->> Hi Ian,
> ->> Thank you for getting back with me.  I'm glad to hear that 
> ->you're interested
> ->> in getting this resolved.  I'll send the patch to you when 
> ->I've finished.
> ->> One major question though:
> ->> 
> ->> You had mentioned that MSVC 7.1 uses a 64 bit streamoff, 
> ->but my research
> ->> seems to contradict this.  My Visual Studio documentation 
> ->said only that:
> ->> "[streamoff's] representation has at least 32 value bits."  
> ->That wasn't very
> ->> definitive, so I checked the return value of 'std::numeric_limits<
> ->> vcl_ios::streamoff >::max()' and it was 2,147,483,647 (2^31 - 1).
> ->
> ->My mistake, I just looked at the include files. It looked to 
> ->me that if 
> ->_WIN64 was defined, streamoff would be 64 bits, otherwise 32 bits. I 
> ->guess that means that the processor/compiler may have 64 
> ->bits, but the 
> ->OS still uses a 32 bit API. When a proper 64 bit Windows API 
> ->is un use, 
> ->then streamoff will go to 64 bits.
> ->
> ->It strikes me that this is a limitation of the C++ standard. I guess 
> ->strictly we need to add a VCL_SIZEOF_STREAMOFF into the config.
> ->
> ->> Now, this isn't to say that solution #1 can't work.  I 
> ->would just also have
> ->> to modify vil_stream_fstream::seek() to do multiple seeks 
> ->when the desired
> ->> position exceeds 'std::numeric_limits< vcl_ios::streamoff 
> ->>::max()'.  The
> ->> only problem I see with this is that the function could end 
> ->up executing as
> ->> many as ~2^32 seeks (for the worst case of passing in a 
> ->desired position of
> ->> 2^63) -- that might be prohibitively slow.  I'm not sure 
> ->where someone would
> ->> get access to a stream that big though (especially on a 32 
> ->bit system), so
> ->> it should be ok.  I.e. this approach would work nicely for 
> ->all reasonable
> ->> cases.  What do you think?
> ->
> ->This would work. I think your worst case scenario is somewhat 
> ->unlikely. 
> ->Anyway if you jsut docuement the problem in the code, and tell users 
> ->that they need to get a C++ implementation that has a 64 bit 
> ->streamoff.
> ->
> ->I'll be happy with your patch if the tests continue to work. I (and I 
> ->guess most users) don't have any code that uses a 
> ->vil_streampos directly 
> ->- It's all hidden in the vil file loaders.
> ->
> ->Ian.
> ->
> ->
> 
>

Hi Ian,
As we discussed previously, I changed the declaration of vil_streampos to be
a 64 bit integer on platforms that support it.  I then resolved all the
compiler warnings that this change caused (ie. Code that used long, int and
vil_streampos interchangeably).  I also verified that all the tests in
vil_test_all still pass after these changes were applied.  The patch for all
of this is attached.  I think these changes are worth adding despite the bad
news that follows.

My idea for getting vil_stream_fstream::seek() to work is not viable.  It
appears as if any attempt to seek past 2GB fail regardless of whether it's
done in one seek call or broken up into multiple seeks.  So, I couldn't
really think of a way to get vil_stream_fstream to work properly with large
files -- at least not while using vcl_fstream under the hood.  

What I did to work around this was create a new class, vil_stream_fstream64,
that uses low level POSIX (or POSIX-like) system calls.  I've attached the
new class files in case anyone wanted to check them out, but they aren't yet
ready for the vxl core.  I don't believe that this class (in its current
state) will compile on any platform except windows.  

I'm not really familiar with the POSIX implementations on all VXL platforms,
so I don't really know what level of effort it will take to make this class
work on all VXL platforms.  Does anyone have any advice on this?

Thanks,
Rob