[phpesp-dev] [ phpesp-Bugs-1387534 ] Export fails to download all the data

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Bugs item #1387534, was opened at 2005-12-21 21:33
Message generated for change (Comment added) made by jfculst
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=108956&aid=1387534&group_id=8956

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Admin
Group: v1.7
Status: Open
Resolution: None
Priority: 5
Submitted By: Neil (neilsourceforge)
Assigned to: Nobody/Anonymous (nobody)
Summary: Export fails to download all the data

Initial Comment:
We upgraded to 1.7.5 from 1.7.2 Monday and today the 
phone rings.

Developers with substantial response sets get a 
partial dat set when Exporting without any feedback 
about not getting all the data. As an example a 
response set of 987 records, 54 fields returns a CSV 
of 123 records all fields. All formats produce the 
same results.

Which parameters influence download limits?

Is there a SELECT or procedure where I can extract all 
responses for a survey directly from the MySQL 
database?

Info:
PHP Information 
Version: 4.3.11 
OS: Linux 
SAPI: apache2handler 
register_globals: No 
magic_quotes_gpc: No 
magic_quotes_runtime: No 
safe_mode: No 
open_basedir: 

PHP Extensions 
dBase: No 
GD: Yes -- bundled (2.0.28 compatible) 
GNU Gettext: Yes 
LDAP: Yes 
MySQL: Yes 
PHP Extension Dir (compiled): /usr ...
PHP Extension Dir (run time): /usr ...

phpESP Settings 
Expected ESP_BASE: /fsys ...
Expected base_url: uw ... 
Loading phpESP.ini.php ...

ESP_BASE: /fsys ...
base_url: https://uw ...
Version: 1.7.5 
Debug: No 

phpESP Language Settings 
GNU Gettext: Real 
default_lang: en_US 
current lang: en_US 
available langs: da_DK, de_DE, el_GR, en_US, es_ES, 
fi, fi_FI, fr_FR, hu_HU, it_IT, ja_JP, nl_NL, pt_BR, 
pt_PT, sv_SE
(da, de, el, en, es, fi, fr, it, ja, nl, pt, sv) 
GNU Gettext test: Passed 
Catalog Open Test: Yes 

PHP Session Test 
session.save_path: /var ...
Counter: 1 

----------------------------------------------------------------------

Comment By: James Coyle (jfculst)
Date: 2006-09-28 11:21

Message:
Logged In: YES 
user_id=1393403

It works! Admin were a bit nervious about leaving the www 
server vunerable extending the timeout from 30 secs to 
600. 
It took over 5 mins to download the 706KB file far too 
long.
php.ini in the script directory should work, but still 
leaves a security vunerability.
Thanks Aaron

----------------------------------------------------------------------

Comment By: Aaron Axelsen (axelseaa)
Date: 2006-09-26 13:45

Message:
Logged In: YES 
user_id=704595

Have you tried increasing the max execution time in php? 
This can be done in either the php.ini or in a .htaccess file.

Depending upon your system speed, a survey with that many
records could take a minute or two to export.

----------------------------------------------------------------------

Comment By: James Coyle (jfculst)
Date: 2006-09-26 11:53

Message:
Logged In: YES 
user_id=1393403

We have produced a survey of 81,509 records and 613 
responses. I am unable to download the CSV file, it times 
out with the error: 
Fatal error: Maximum execution time of 40 seconds exceeded 
in /web2/science/survey/phpesp/admin/include/lib/esprespons
e.inc on line 289
=====================
I tried the header solution mentioned but it fails also. 
Is there a ready made patch I could insert?, I am using 
ver 1.7.5

----------------------------------------------------------------------

Comment By: Neil (neilsourceforge)
Date: 2006-01-05 21:00

Message:
Logged In: YES 
user_id=1181273

We have discovered details that further an explanation of 
why our attempts to Export respondent data fails (silently) 
for large data sets. Our recent tests are against release 
1.8 of phpESP.

We implement phpESP using SSL. Internet Explorer 6 fails to 
export any data with the following objection:
"Internet Explorer cannot download ...header ...
Internet Explorer was not able to open this site..."

By adjusting the file download.inc and adding these header 
statements the download appears to function.

header("Content-Type: $mime");
header("Cache-Control: post-check=999, pre-check=999");
header("Pragma: cache");
header("Content-Disposition: attachment; filename={$name}");
header("Content-Length: ".strlen($data));

But fails for relatively large data sets. The variable 
$data seems not to exist. By suppressing the 'Content-
Length' directive altogether exports appear to operate 
fully, our smallest and our largest respondent sets.

I would like to suggest that support for IE 6 to an SSL 
enabled server be added to phpESP.

.../neil patterson

----------------------------------------------------------------------

Comment By: Neil (neilsourceforge)
Date: 2005-12-22 21:21

Message:
Logged In: YES 
user_id=1181273

IE 6, latest Firefox and Netscape all fail to extract the 
entire data set. 

----------------------------------------------------------------------

Comment By: Matthew Gregg (greggmc)
Date: 2005-12-22 20:44

Message:
Logged In: YES 
user_id=14116

If you can download to the server then it might be a browser
timeout issue can you confirm the lack of responses on
various browsers(IE/Firefox)?

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2005-12-22 20:27

Message:
Logged In: NO 

We reinstalled 1.7.2 and the larger data sets also fail on 
export.

There are no errors in the web logs related to the download 
issue.

max_execution_time and max_input_time are 60 seconds, and 
the memory_limit is 24M.

----------------------------------------------------------------------

Comment By: Neil (neilsourceforge)
Date: 2005-12-22 18:40

Message:
Logged In: YES 
user_id=1181273

I can respond to two of your questions and will respond to 
the rest when I have the information.

I did no test this against 1.7.2 though large downloads 
will be a test of all future releases. 1.7.2 has been 
active for several months and no complaints; I would infer 
that 1.7.2 was OK.

I will respond to the Apache log review later. Nothing on 
the client screen and nothing in the Application or System 
event logs on the client computer.

I can successfully download to the server. All data appear 
available. It is only the local downloads that fail. The 
server download implies the data are valid, no incomplete 
responses.

I will confirm the memory and execution limits later.

----------------------------------------------------------------------

Comment By: Matthew Gregg (greggmc)
Date: 2005-12-22 15:24

Message:
Logged In: YES 
user_id=14116

Results exported without a problem in 1.7.2?  Do you have
any errors in Apache's log file when the export fails or in
the exported data? I have a 4000 response survey that
manages to export completetly. 

Is it possible that the survey has incomplete
responses(save/resume survey)?

You might need to increase the memory and execution limts
for PHP.  

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=108956&aid=1387534&group_id=8956