Re: [limesurvey-developers] Problem: Limitations by Suhosin & max_input_vars in actual PHP configur
The leading Open Source survey tool
Brought to you by:
c_schmitz
Menu
▾
▴
Re: [limesurvey-developers] Problem: Limitations by Suhosin & max_input_vars in actual PHP configurations
|
From: Thomas W. <tom...@gm...> - 2012-03-05 21:31:11
|
A quick clarification. EM only includes <input> fields for prior variables if they are definitely used in the conditions on the page - so it isn't including all prior variables. If all prior variables are being included the page, it isn't because of EM - we could track that down and squash it. Other things that can be done: (1) Remove all of the new relevance* variables and replace with a single JSON array - there is one per question and group (plus one per SGQA if you do sub-question level relevance processing). This would require small changes to em_javascript.js (2) Should be able to remove another 50% of input - things that are reference by id="sgqa" in jQuery, statements, but aren't the javaSGQA variable that is POSTed and processed to save the data - so just remove their name="" attribute and they won't be posted. - seems like this would be safer than trying to use disabled=disabled, since wouldn't that have side effects? /Tom On Mon, Mar 5, 2012 at 1:56 PM, Tac Tacelosky <ta...@gm...> wrote: > Good point. I'm thinking that this is just a stop-gap measure for > 1.9.x, and very much hoping that for 2.x we'll be using field names > based on the question code rather than SGQA, it will make javascript > and EM coding sooooooo much more readable. EM is already moving in > that direction. > > Adding "disabled='disabled'", as Denis suggests, to all the fields not > actually used in the POST is probably the easiest first step with the > least amount of disruption. > > We could also disable all fields that are empty, as long as the > processing could handle the fields being missing and defaulted > appropriately. > > Tac > > On Mon, Mar 5, 2012 at 1:50 PM, Tony Partner <tpa...@pa...> > wrote: > > I have concerns about modifying 'fieldnames'. I think there is lots of > > custom JavaScript out there (mine and others) that uses this input to > find > > the survey and group IDs which are then used in SGQA identifiers. > Removing > > the Survey and group IDs would render those scripts useless. > > > > T. > > > > -----Original Message----- > > From: Tac Tacelosky [mailto:ta...@gm...] > > Sent: March-05-12 1:23 PM > > To: lim...@li... > > Subject: Re: [limesurvey-developers] Problem: Limitations by Suhosin & > > max_input_vars in actual PHP configurations > > > > It appears to me (and I could be completely wrong) that a lot of the > hidden > > fields exist to help in the javascript execution, and aren't required to > be > > submitted. > > > > A couple of ideas for reducing the sheez volume of the submitted data: > > > > (1) In the fieldnames hidden var, which looks like this: > > > > <input type='hidden' name='fieldnames' > > > value='82556X23X493sq1#0|82556X23X493sq1#1|82556X23X493sq2#0|82556X23X493sq2 > > > #1|82556X23X493sq3#0|82556X23X493sq3#1|82556X23X493sq4#0|82556X23X493sq4#1|8 > > > 2556X23X493sq5#0|82556X23X493sq5#1|82556X23X494sq1_1|82556X23X494sq1_2|82556 > > > X23X494sq1_3|82556X23X494sq2_1|82556X23X494sq2_2|82556X23X494sq2_3|82556X23X > > > 494sq3_1|82556X23X494sq3_2|82556X23X494sq3_3|82556X23X494sq4_1|82556X23X494s > > > q4_2|82556X23X494sq4_3|82556X23X494sq5_1|82556X23X494sq5_2|82556X23X494sq5_3 > > > |82556X23X495sq1_1|82556X23X495sq1_2|82556X23X495sq1_3|82556X23X495sq2_1|825 > > > 56X23X495sq2_2|82556X23X495sq2_3|82556X23X495sq3_1|82556X23X495sq3_2|82556X2 > > > 3X495sq3_3|82556X23X495sq4_1|82556X23X495sq4_2|82556X23X495sq4_3|82556X23X49 > > > 5sq5_1|82556X23X495sq5_2|82556X23X495sq5_3|82556X23X496sq1|82556X23X496sq2|8 > > > 2556X23X496sq3|82556X23X496sq4|82556X23X496sq5|82556X23X497sq1|82556X23X497s > > > q2|82556X23X497sq3|82556X23X497sq4|82556X23X497sq5|82556X23X498sq1|82556X23X > > > 498sq2|82556X23X498sq3|82556X23X498sq4|82556X23X498sq5|82556X23X499sq1|82556 > > > X23X499sq2|82556X23X499sq3|82556X23X499sq4|82556X23X499sq5|82556X23X500sq1|8 > > > 2556X23X500sq2|82556X23X500sq3|82556X23X500sq4|82556X23X500sq5|82556X23X501s > > > q1|82556X23X501sq2|82556X23X501sq3|82556X23X501sq4|82556X23X501sq5|82556X23X > > > 501other|82556X23X502sq1|82556X23X502sq1comment|82556X23X502sq2|82556X23X502 > > > sq2comment|82556X23X502sq3|82556X23X502sq3comment|82556X23X502sq4|82556X23X5 > > > 02sq4comment|82556X23X502sq5|82556X23X502sq5comment|82556X23X502other|82556X > > 23X502othercomment|82556X23X503other|82556X23X503' > > id='fieldnames' /> > > > > get rid of everything before the second X, and send in the survey id > > independently. The survey id is the same, and the group id can be > looked up > > from the question id, so it's unnecessary to send. > > > > (2): Change the field names as described above, e.g. X502other. That > will > > get rid of the redundant survey id and the unnecessary group id. > > > > (3) Move any of the hidden variable beginning with 'java' to outside the > > form: > > > > <input type="hidden" name="java82556X23X493sq1#1" > > id="java82556X23X493sq1#1" value="" /> > > > > If they're not needed by the POST. > > > > Just some ideas. > > > > Tac > > > > > > On Mon, Mar 5, 2012 at 1:01 PM, Carsten Schmitz > > <car...@li...> wrote: > >> > >> > >> Hello! > >> > >> Good idea! > >> But the problem with serializing all the inputs (as the only > > optimiziation) that the it would hit another limit pretty > > fast:max_value_length (Suhosin again). > >> We can already see that in the quick translation feature of the admin, > > which does exactly that and managed to hit that limit in the past. > >> > >> I think that serialization would be a great optimization though after we > > reduced the number of inputs in general. > >> > >> Regards > >> > >> Carsten > >> > >> Am 05.03.2012 18:11, schrieb Tac Tacelosky: > >> > >> By coincidence, I was discussing a related issue probably while you were > > typing this. The related issue was about using LimeSurvey to do Amazon > > Mechanical Turk tasks, which would require that we first submit the > > responses to LS (via Ajax) and then do the actual submit to the Turk > site. > > And it's tricky because I really just wanted to serialize the form, do > the > > Ajax submit to LS, and then in the return do the submit to Turk. But the > > way the javascript works makes this pretty awkward, so I figured I'd wait > > until Version 2 to address it. > >> > >> I assume you're talking about a 1.9.x fix for this issue, not waiting > > until Version 2. > >> > >> Anyway, brainstorming about a solution to the max_input_vars issue, what > > do you think about serializing the form and the serialized form data as a > > single JSON string, e.g. form_data_json? That would mean that index.php > > would need to deserialize it. > >> > >> So the flow could be > >> > >> User clicks on submit > >> JS is run to validate the form. > >> If valid, serialize the form data to a hidden field (data_json) in > >> other form, and POST that form OR > >> serialize the form data to a hidden field (data_json), > >> disable the rest of the fields, and POST the form > >> > >> If we did that, it'd be nice to submit some data about the save to the > > next_url, like the survey_id and the response_id -- that would allow > custom > > reporting, or even doing a post and submitting all of the data about the > > response (to save a database lookup). I'd use it to then directly to > Turk. > >> > >> Anyway, I think that's one solution to reducing the max_input_vars. > >> > >> Tac > >> > >> On Mon, Mar 5, 2012 at 11:51 AM, Carsten Schmitz > > <car...@li...> wrote: > >>> > >>> Hello everyone, > >>> > >>> this was discussed recently but this issue just became a problem so I > >>> would like to discuss it here. > >>> > >>> So far the dreaded Suhosin security module for PHP had been only used > >>> in a minor (but rising) number of server installations. > >>> The biggest problem of this PHP module is that it limits the number > >>> of variable that you can POST on a single page. > >>> > >>> Most of the times people could work around it by setting the module > >>> to simulation mode. Recently a new 'inconvenience' was added in PHP > >>> 5.3.9 with the parameter max_imput_vars which basically does the same > >>> as the Suhosin module. > >>> > >>> You would think that's no big deal because 5.3.9 has been released > >>> only a short time ago and providers take some time to upgrade. > >>> Far from true! Due to recent exploit (which uses a huge number of > >>> POST vars to open a security hole) the biggest distributions (like > >>> Ubuntu, Debian and others) backported max_input_vars AND the Suhosin > >>> module to all long-term-support (LTS) server releases. > >>> > >>> That means that most of the users who want to use LimeSurvey now will > >>> run into this problem if their survey get alittle bigger unless we > >>> do something about it. > >>> > >>> Thomas (tmswhite) already removed some of the POST vars (which became > >>> obsolete due to EM) embedded on each survey page and so was able to > >>> reducing the number of POST vars by an average of 33%. > >>> This really helps alot. Now with the standard setting of PHP this > >>> would result in almost 500 responses to be submitted on one page > >>> without a problem - but.... > >>> when using conditions currently all previous responses are embedded > >>> on a page as hidden inputs which breaks this limit easily. > >>> > >>> This seems to be very inefficient, not only because we reach the > >>> max_input_vars limit, but also we submit alot of unnecessary data to > >>> the server on each page submit and the page itself becomes huge. > >>> > >>> I discussed this with Jason and we came up with two solutions, but we > >>> are not sure if these are easily doable: > >>> > >>> 1.) Scan all conditions/EM/micro-tailoring formulas on the page and > >>> only embed values that are really needed on that page. > >>> 2.) Do not embed values anymore by hidden <INPUT> fields but by > >>> dynamic JS or some other form of HTML that is NOT sent to the server. > >>> > >>> If you have any better ideas please let us know. Fact is, this > >>> problem needs to be fixed as soon as possible - otherwise we will > >>> have alot more user with problems on our hands very soon. > >>> > >>> > >>> Best regards from Hamburg/Germany > >>> > >>> Carsten Schmitz > >>> > >>> LimeSurvey Project Leader > >>> car...@li... > >>> > >>> http://www.limesurvey.org > >>> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> --------- Try before you buy = See our experts in action! > >>> The most comprehensive online learning library for Microsoft > >>> developers is just $99.99! Visual Studio, SharePoint, SQL - plus > >>> HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when > you > > subscribe now! > >>> http://p.sf.net/sfu/learndevnow-dev2 > >>> _______________________________________________ > >>> limesurvey-developers mailing list > >>> lim...@li... > >>> https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > >> > >> > >> > >> > >> ---------------------------------------------------------------------- > >> -------- Try before you buy = See our experts in action! > >> The most comprehensive online learning library for Microsoft > >> developers is just $99.99! Visual Studio, SharePoint, SQL - plus > >> HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you > > subscribe now! > >> http://p.sf.net/sfu/learndevnow-dev2 > >> > >> > >> > >> _______________________________________________ > >> limesurvey-developers mailing list > >> lim...@li... > >> https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > >> > >> > >> > >> ---------------------------------------------------------------------- > >> -------- Try before you buy = See our experts in action! > >> The most comprehensive online learning library for Microsoft > >> developers is just $99.99! Visual Studio, SharePoint, SQL - plus > >> HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you > > subscribe now! > >> http://p.sf.net/sfu/learndevnow-dev2 > >> _______________________________________________ > >> limesurvey-developers mailing list > >> lim...@li... > >> https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > >> > > > > > ---------------------------------------------------------------------------- > > -- > > Try before you buy = See our experts in action! > > The most comprehensive online learning library for Microsoft developers > is > > just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro > > Style Apps, more. Free future releases when you subscribe now! > > http://p.sf.net/sfu/learndevnow-dev2 > > _______________________________________________ > > limesurvey-developers mailing list > > lim...@li... > > https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > > > > > > > > > > ======= > > Email scanned by PC Tools - No viruses or spyware found. > > (Email Guard: 7.0.0.21, Virus/Spyware Database: 6.18900) > > http://www.pctools.com/ ======= > > > > > > > ------------------------------------------------------------------------------ > > Try before you buy = See our experts in action! > > The most comprehensive online learning library for Microsoft developers > > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > > Metro Style Apps, more. Free future releases when you subscribe now! > > http://p.sf.net/sfu/learndevnow-dev2 > > _______________________________________________ > > limesurvey-developers mailing list > > lim...@li... > > https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > > > ------------------------------------------------------------------------------ > Try before you buy = See our experts in action! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-dev2 > _______________________________________________ > limesurvey-developers mailing list > lim...@li... > https://lists.sourceforge.net/lists/listinfo/limesurvey-developers > |