Re: [Treebase-devel] harvesting

[William P. <wil...@ya...>]

On Jan 25, 2012, at 12:16 AM, William Piel wrote:

> Both TreeBASE's tallest dataset (~3,000 taxa) and it's widest dataset (~110,000 characters), download just fine

Actually, I spoke too soon. Mattison just noticed that a lot of nexml requests are getting hung up after he asked for "S12156?format=nexml". 

On Jan 25, 2012, at 2:41 PM, Mattison Ward wrote:
> FYI - I tried this url to make sure everything was working
> http://purl.org/phylo/treebase/phylows/study/TB2:S12156?format=nexml
> 
> Then I tried this -
> http://purl.org/phylo/treebase/phylows/study/TB2:S12329?format=nexml
> 
> Neither of these recent entries started downloading in the 5 or so
> minutes that I waited.
> 
> Other downloads seem to be working ok.

It turns out that study S12156 has two matrices -- one that is 876,159 characters wide, and the other that is 352,120 characters wide, so a NeXML download for this study is asking for 1,228,279 characters. 

Oddly enough, the same data downloaded as "?format=nexus" works fine... so the problem is with the efficiency of generating the NeXML serialization. But another strange effect is that all subsequent requests for NeXML files are also blocked -- i.e. once you clog the system with a massive NeXML request, others get hung too. 

So, for example, S1205 works fine on dev: 

http://purl.org/phylo/treebase/dev/phylows/study/TB2:S1205?format=nexml

... but remains clogged on production, due to (I guess) several open requests for S12156. 

Rutger -- maybe before trying to suck down the database, you should first look into why our NeXML serializations are hanging? I notice that some things make NeXML much more verbose -- e.g. when CHARSETs are present -- could those be a problem? 

bp



17 current requests        Details      
Thread
Request
Elapsed time (ms)
Mean time (ms)
Cpu time (ms)
Mean cpu time (ms)
Hits sql
Mean hits sql
Time sql (ms)
Mean time sql (ms)
Executed method
http-8080-Processor6	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,929,163	6,418	1,711	476	9	152	23,009	2,104	java.util.HashMap.getEntry(HashMap.java:364)
http-8080-Processor17	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,794,021	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor50	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,715,306	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor9	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,628,908	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor16	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,420,516	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor7	
 /search/downloadAStudy.html?id=12329&format=nexml GET
 3,386,509	6,418	738	476	9	152	335	2,104	java.util.HashMap.getEntry(HashMap.java:364)
http-8080-Processor3	
 /search/downloadAStudy.html?id=12329&format=nexml GET
 3,331,477	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor49	
 /search/downloadAStudy.html?id=12329&format=nexml GET
 3,239,040	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor15	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,045,588	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor10	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 3,013,936	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor1	
 /search/downloadAStudy.html?id=12156&format=nexml GET
 2,968,063	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor19	
 /search/downloadAStudy.html?id=10149&format=nexml GET
 2,934,114	6,418	0	476	6	152	260	2,104	java.util.HashMap.getEntry(HashMap.java:364)
http-8080-Processor20	
 /search/downloadAStudy.html?id=12329&format=nexml GET
 2,863,803	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor4	
 /search/downloadAStudy.html?id=736&format=nexml GET
 2,797,320	6,418	0	476	11	152	439	2,104	java.util.HashMap.getEntry(HashMap.java:364)
http-8080-Processor47	
 /search/downloadAStudy.html?id=1205&format=nexml GET
 1,477,133	6,418	0	476	9	152	151	2,104	java.util.HashMap.getEntry(HashMap.java:364)
http-8080-Processor8	
 /search/downloadAStudy.html?id=1205&format=nexml GET
 955,541	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)
http-8080-Processor43	
 /search/downloadAStudy.html?id=1205&format=nexml GET
 698,686	6,418	0	476	0	152	0	2,104	java.lang.Object.wait(Native Method)