[OpenSTA-devel] Re: [ opensta-Bugs-1083975 ] Concatenating large SCL strings can cause compiler to crash

[Daniel S. <da...@Op...>]

Thierry Boullet wrote:
> Thanks Dan for this new release.  This is great news.

Well it was, in no small part, made possible by work put in by you,
so I return the Thanks to you and agree that is great news.  It will
be made greater if I can maintain the momentum and get another
release (1.4.4) out in the next month or so...

> I reproduced this bug during the use of OpenSTA 1.4.2 with large
> viewstate.

As I hinted in the bug report - I suspected this bug was not dead,
I'm glad that you have confirmed this and hopefully we can reproduce
and fix it properly.

It is probably worth looking into bug#1083968:
  http://sourceforge.net/tracker/index.php?func=detail&aid=1083968&group_id=10857&atid=110857
while we are investigating this area.  This problem is where the
compiler gives the MAXSTRLEN error even though the string is less
than 64K in length...  this problem was the reason I just didn't
split the strings into 64K chunks for my fix for bug#637865 - and
probably why I ended up not fixing that properly...

> I made additional tests, here the description and the results:

> OS : Windows XP Pro SP2 with last updates
> 
> SCRIPT_65850_142 : script captured with 1.4.2 containing a literal 
>   string with 65850 chars (body of a POST request, 1 block)
> 
> SCRIPT_65850_143 : script captured with 1.4.3 containing a literal 
>   string with 65850 chars (3 blocks with 2 concatenations)
> 
> SCRIPT_96970_142 : script captured with 1.4.2 containing a literal 
>   string with 96970 chars (body of a POST request, 1 block)
> 
> SCRIPT_96970_143 : script captured with 1.4.3 containing a literal 
>   string with 96970 chars (3 blocks with 2 concatenations)
> 
> SCRIPT_65850_142 and SCRIPT_96970_142 compiled in Script Modeler
>   1.4.2  or 1.4.3 => error maxstrlen (that's normal)
> SCRIPT_65850_143 compiled in Script Modeler 1.4.2 or 1.4.3 =>
>    success (Is this normal since total size of the string > 65535?)

That is "normal" and has been the previous workaround for the large
BODY problems of the past.

> SCRIPT_96970_143 compiled in Script Modeler 1.4.2 or 1.4.3 => crash
>   of scl.exe

This is definitely bug#1083975 I experienced and logged and then
couldn't reproduce.

> I built another version of OpenSTA which is based on the 1.4.3 with
> the following modifications:
> - file \src\Architecture\Scl\scl_limits.h:
>   #define MAX_VARIABLE_SIZE 262143 (Maximum size of a variable or record)
>   #define MAX_LITERAL_SIZE 262143 (Maximum size of a character literal or constant)
> - file \src\Architecture\Scl\scl_tpacom.h :
>   #define MAX_CONSTANT_SIZE 262143 (Max size of constants area)
>   #define CHRBUFSIZ 262144 (Character string storage)
> 
> Role of MAX_VARIABLE_SIZE :
> for example : CHARACTER*70000 STR_TMP
> => error (modoutrange), Character Modifier Out of Range. Valid Range 1 to 65535
> 
> Role of MAX_LITERAL_SIZE :
> for example : "A very very long sequence of chars more than 64K"
> => error (maxstrlen), Maximum String Length Limit Exceeded
> 
> Role of MAX_CONSTANT_SIZE :
> for example : definition of many large constants (> 64K)
> => error (consoverflow), Constants Data Area Overflow

This is an odd one.  So the MAX_CONSTANT_SIZE is the limit of space
available for all the constants together?  I don't remember that
being how it is documented.

> Role of CHRBUFSIZ  :
> for example : if a string is > CHRBUFSIZ, crash of scl.exe

This tells me that what we should have is something like:
  #define MAX_VARIABLE_SIZE X
  #define MAX_LITERAL_SIZE X
  #define CHRBUFSIZ X*N
Where N give us the room to play around.
And obviously we need some sensible behavior when CHRBUFSIZ is
overflowed.

I also want to know where bug#1083968 comes into play - my thought
is that not all content is equal and further encoding will increase
the actual size of a string... ?

What I don't understand is why I couldn't reproduce this problem if
it was simply a matter of overflowing CHRBUFSIZ!  I was able to
record and compile some HUGE file uploads during my testing - it was
slow but it worked.
 
> The constant MAX_STRING_LEN in the file 
> \src\Architecture\Scl\scl_comdef.h is not used.

Nice - I guess we just need to get rid of this then.

> With this version, results are :
> 
> SCRIPT_65850_142 and SCRIPT_96970_142 compiled in Script Modeler => success
> SCRIPT_65850_143 compiled in Script Modeler => success
> SCRIPT_96970_143 compiled in Script Modeler => success
> 
> The use of variables > 64k functions correctly (DOM Addressing, 
> encoding, use in the body of a POST).

I know of a few people who'd really like to get their hands on your
new build... ;-)

> I think that it is a track to exceed the limit of the 64K when that
> is really necessary.

Maybe in the case of providing POST bodies - but this makes for
really ugly scripts.  It would be nicer to be able to take the 
POST body from a file or something.

My one worry about increasing the maximum size of a variable is just
really moving the goal posts - plus if people have these large
variables they are likely to use them without thought and then
complain about runtime memory usage without a thought to it being
there own fault.  I can see large variables being of real use in
functional testing but in the performance testing that is supposed to
be the focus of OpenSTA they really ought to be avoided and
discoraged.

IMHO the real solution to interrogating returned HTTP data larger
than 64K is a more flexible identifier addressing mechanism.  This
could provide a full solution whilst still being performance oriented
...

> The new limits of 256K that I used are arbitrary.  These limits
> should be configurable.

Do you mean easily configurable at the compilation level - or
something that can be changed from the user level?

> For example to detect size too large of POST request body, which
> can be the cause of bad performances.

One thing I'd like to see is runtime warnings generated everytime
content that is too big is assigned to a character variable and
needs to be truncated.  I think this would be really useful, although
I'm not sure if it is what you meant with this last sentence.

> I will add a comment for the bug 1083975.

Can you attach the script that reproduces the crash to the problem
report?  I'm not sure if you can as a normal user... if you can't,
then I think it's about time we got you added as a member of the
OpenSTA SF project... if you want?

It would be great to get this and the other new issue fixed quickly
and then we can make a new release with the updated build
environment and these fixes out and heavily beta'd so 1.4.4 can
happen in the next couple of months.

Cheers
/dan
-- 
Daniel Sutcliffe <Da...@Op...>
  OpenSTA - part-time caretaker - http://OpenSTA.org/