consequences of increasing Max Record Size

Help
2010-04-01
2013-06-12
  • Nathan Parrish

    Nathan Parrish - 2010-04-01

    I just discovered a problem in our STAX job processing mechanism where things fall apart when some job attempts to log > 100K characters.  Frankly I'm rather surprised I did not hit this limitation much earlier, but in any case it seems rather clear that it's the log service Max Record Size setting causing the issue. 

    As usual, while recognizing an opportunity to overhaul our test script output handling, I'd rather kick this can a bit further down the road.  Accordingly, I'd appreciate any caveats on setting this value (much) higher.  Setting it to 1M would probably do, but at the same time it's not at all far-fetched to imagine a test generating 5MB or more;  I'd just like some reasons why it'd be a particularly bad idea to go for broke and set the value to something like 10 or 100MB. 

    thanks,
    nathan

     
  • Sharon Lucas

    Sharon Lucas - 2010-04-01

    The Log service by default truncates a single message logged to a STAF log file at 100,000 characters.  You can change this default by specifying the MaxRecordSize parameter when registering the LOG service in your STAF.cfg file.  You can also dynamically change it while STAFProc is running by submitting a "SET MAXRECORDSIZE <Size>" request to the LOG service.

    The STAX User's Guide includes the following note in the "STAX Logging" section at http://staf.sourceforge.net/current/STAX/staxug.html#Header_StaxLogging that says:

    Note: If a STAX job contains a <script> element which consists of more than 100K of Python code, you may want to increase the maximum log record size on the STAX service machine. When a STAXPythonEvaluationError signal is raised, a message is written to the STAX job log which includes all the Python code contained in the <script> element where the error occurred, followed by the line number and a description of the problem. If you do not specify a larger maximum record size for the Log service, the message logged will be truncated at 100,000 bytes (the default maximum record size for the Log service). To increase the maximum log record size, see the "Log Service" section in the STAF User's Guide for a description of how to specify a MAXRECORDSIZE parameter when registering the Log Service.

    We typically increase the LOG service's MAXRECORDSIZE to 1000000 characters (1M) on our STAX service machines.

    The possible downsides to setting the LOG service's MAXRECORDSIZE to a much higher value are:
    - It takes more time to query large log files.
    - Logging a very long message (e.g. > 1M) might not be very readable via a LOG QUERY request.
    - If someone accidently logged a very long message when they didn't really mean to, the message won't get truncated.

    But, if those aren't issues for you, you can assign it to be any value you want < 4G.

     
  • Sharon Lucas

    Sharon Lucas - 2010-04-01

    It might be a good idea for us to have the STAX service to automatically increase the LOG service's MAXRECORDSIZE to 1,000,000 (if it is set to the default value of 100,000).

    We previously did something similar to have the STAX service automatically increase the MAXQUEUESIZE to 1000 because the default MaxQueueSize is 100 which can be too small for STAX job handle queues.

    Can you open a feature request for this?

     
  • Nathan Parrish

    Nathan Parrish - 2010-04-01

    Thanks,

    I just realized I don't know how to set service settings for "built-in" services.  That is, I set parameters for STAX, and our own home-brewed service, but these are external and thus launched explicitly in STAF.cfg.  How do I set parameters for the LOG service which is not explicitly mentioned in STAF.cfg? 

     
  • Sharon Lucas

    Sharon Lucas - 2010-04-01

    You have to register the LOG service in the STAF.cfg file for your STAX service machine.  The STAF User's Guide talks about how to do this in sections "8.8.2 Registration" and "8.8.3 Parameters".  For example:

    service LOG library STAFLog parms "MAXRECORDSIZE 1000000"

     
  • Nathan Parrish

    Nathan Parrish - 2010-04-01

    just to be clear, this simply overrides (rather than conflicts with) what is apparently an implicit LOG service? 

     
  • David Michael Bender

    Yes.  If you don't  explicitly register the Log service in your STAF.cfg File, the default service loader service (via the "serviceloader library STAFDSLS" statement in your STAF.cfg file) will automatically register the Log service when the first Log service request is submitted to the machine.  But in that scenario the Log service is always registered with the default options.  So, to use any non-default options, you must explicitly register the Log service with the options you want.

     

Log in to post a comment.