Re: [Mesa3d-dev] Mesa (master): mesa: Fix unsigned comparison.

[Brian P. <br...@vm...>]

Michel Dänzer wrote:
> On Thu, 2010-03-04 at 02:00 -0800, Vinson Lee wrote: 
>> Michel, thanks for spotting this.
>>
>> I've reverted the bad commit. Please go ahead and submit your correct fix.
> 
> Actually, I wonder if something like the below isn't needed to avoid any
> undesired effects from integer overflows.
> 
> 
> diff --git a/src/mesa/main/api_validate.c b/src/mesa/main/api_validate.c
> index 326ad6f..8460265 100644
> --- a/src/mesa/main/api_validate.c
> +++ b/src/mesa/main/api_validate.c
> @@ -130,6 +130,7 @@ check_index_bounds(GLcontext *ctx, GLsizei count, GLenum type,
>     struct _mesa_prim prim;
>     struct _mesa_index_buffer ib;
>     GLuint min, max;
> +   GLint64 min64, max64;
>  
>     /* Only the X Server needs to do this -- otherwise, accessing outside
>      * array/BO bounds allows application termination.
> @@ -146,9 +147,12 @@ check_index_bounds(GLcontext *ctx, GLsizei count, GLenum type,
>     ib.obj = ctx->Array.ElementArrayBufferObj;
>  
>     vbo_get_minmax_index(ctx, &prim, &ib, &min, &max);
> +   min64 = min;
> +   min64 += basevertex;
> +   max64 = max;
> +   max64 += basevertex;
>  
> -   if (min + basevertex < 0 ||
> -       max + basevertex > ctx->Array.ArrayObj->_MaxElement) {
> +   if (min64 < 0 || max64 > ctx->Array.ArrayObj->_MaxElement) {
>        /* the max element is out of bounds of one or more enabled arrays */
>        _mesa_warning(ctx, "glDrawElements() index=%u is "
>  		    "out of bounds (max=%u)", max, ctx->Array.ArrayObj->_MaxElement);

I'd be happy with a simple cast for now.  One of these days we'll need 
to be concerned with 4GB buffers (or 2GB buffers and signed 
arithmetic) but we're not quite there yet.  There'd probably be other 
issues to fix.  The buffer object code will need to be audited.

Generally, when fixing signed / unsigned comparison warnings I cast 
the unsigned value to signed, Vinson.

-Brian