Re: [limesurvey-developers] DB Storage Engine Project Proposal

[Marcel M. (Limesurvey) <mar...@li...>]

Thanks, Jason, for summing up our IRC discussion.

I'm no database expert but from my point of view having two or more ways 
to store and retrieve data (the old LS1.x style and a more normalized 
databased) doesn't make much sense and makes maintaining the code more 
difficult.
What about some research on Limesurvey usage regarding number of answers 
and survey size? Carsten should have some data from Limeservice and we 
can also create a simple survey and link it at limesurvey.org. The goal 
is to find out more about maximum survey size and maximum number of 
answers to a survey. Once we have some data we can try to set up a 
normalized database and code a simple script to fill it with the max 
numbers of answers and then test performance when retrieving data to 
find out if we might run into problems with this kind of data storage. 
You don't need to code much for testing this so I consider this a better 
way than having an extremely flexible API which has to handle different 
database schemas.

What do the others think about it?

Best regards,
Marcel



Jason Cleeland schrieb:
>
> Hi everyone,
>
> We had a chat about this proposal on IRC the other day and I said that 
> I would draft up a project proposal.
>
> The idea is that this might become the central project for this year, 
> and that we may even consider having multiple students working on it 
> together.
>
> Would those who are particularly interested mind having a look at this 
> and letting me know what they think before I put it onto the wiki?
>
> Thanks,
>
> Jason Cleeland
>
> *Project Proposal*
>
> *Survey Database Storage Engine *
>
> Develop an extensible plugin survey storage engine for LimeSurvey 2.
>
> *Description*
>
> Other than survey design, everything that happens in LimeSurvey 
> centres around the ability to save responses to questions to a 
> database, and retrieve the responses again when required. Historically 
> these functions have been incorporated within the various functions of 
> LimeSurvey locking LimeSurvey in to a single methodology for data 
> storage and retrieval and making future modifications and improvements 
> difficult if not possible.
>
> For this project we are looking for a student to write a plugin module 
> or class that will act as a mediator between LimeSurvey functions and 
> the database for such things as the processing of surveys, the 
> statistics functions, the export functions and the user management 
> functions.
>
> The project would expose a consistent set of functions to LimeSurvey 
> for storing and retrieving survey responses.
>
> It would connect seamlessly with the cakephp database framework to 
> save, delete, retrieve or modify survey responses. By minimising the 
> interaction with the built-in cakePHP framework it would ensure future 
> changes in coding frameworks could be managed with minimum difficulty.
>
> It would be extensible in the sense that it could manage more than one 
> database and field naming methodology.
>
> *Rationale*
>
> Because the interface with the data storage system is so critical to 
> practically every other function in LimeSurvey we want to develop a 
> stand-alone module that is transportable and extensible for the 
> future. And by keeping the logic of data and survey storage separate 
> from the rest of LimeSurvey’s functions we will the ease with which 
> LimeSurvey can be developed in the long term.
>
> *Implementation*
>
> This is such an important project that we would consider multiple 
> students undertaking the task. In conjunction with a strong 
> understanding of data structures, php coding and api principles the 
> students would also have to display a capacity to work in a team and 
> with the broader community.
>
> The implementation would focus on becoming a plugin/addon module under 
> CakePHP which uses the default CakePHP database systems to the 
> absolute minimum. In a well implemented version of this proposal only 
> simple SQL inserts, queries and updates would be required. All other 
> logic would be managed by the engine.
>
> *Field Naming and Database Extensibility Methodologies*
>
> An ongoing debate in the LimeSurvey community has been the method in 
> which the database stores survey results. In LimeSurvey 1 a flat 
> database schema was used whereby once the survey designer had finished 
> setting up the survey a database table was created with a field for 
> every individual question/answer combination. The advantages of this 
> database schema were speed of retrieval for reporting and exporting 
> and searching. The disadvantages were that the survey system had to 
> ‘freeze’ the structure in place once the table was created (therefore 
> the activate/de-activate rules in LimeSurvey 1). The single table 
> method doesn’t allow easy storage of metadata for individual 
> question/answer combinations (such as timestamps). Nor does it lend 
> itself to surveys with enormous quantities of question/answer 
> combinations – sometimes running up against database limitations 
> regarding the number of fields per table.
>
> An alternative is a ‘normalised’ database topology where a single 
> table is created to store all survey responses – one database row per 
> question/answer combination. A related table would store metadata such 
> as timestamps and data type. The advantage to this methodology is the 
> increased flexibility (extensible) which would allow modification of 
> surveys at any stage even after data has been collected. The 
> disadvantages are that searching and compilation of data is 
> complicated and resource intensive.
>
> In a perfect world LimeSurvey would allow users to choose either 
> methodology – either at an installation level or even on a per-survey 
> level.
>
> Having a database storage engine effectively acting as an API would 
> allow this functionality to be built, but the choice of database 
> topology would be invisible to the related LimeSurvey functions that 
> require data.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> ------------------------------------------------------------------------
>
> _______________________________________________
> limesurvey-developers mailing list
> lim...@li...
> https://lists.sourceforge.net/lists/listinfo/limesurvey-developers
>