Re: [Exist-open] http client hanging

[Winona S. <wsa...@gm...>]

Hi Adam,
Have you had a chance to look at this? This is mission critical for 3 of my
current projects, and one of them also requires the new ICU4j integration
for searching. I would love to see a fix for this in the next release if
possible.

Is there anything I can do on my side to help debug?

Thanks,
-Winona


On Sun, Jan 14, 2018 at 10:05 AM, Winona Salesky <wsa...@gm...> wrote:

> Hi Adam,
> There was a typo in the fist script above. Sorry.
> You can run this:
>
> xquery version "3.1";
>
> import module namespace http="http://expath.org/ns/http-client";
>
> declare function local:get-zotero-data($url){
>    http:send-request(<http:request http-version="1.1"
> href="{xs:anyURI($url)}" method="get">
>                               <http:header name="Connection"
> value="close"/>
>                       </http:request>)
> };
>
> declare function local:process($url){
>     let $items := local:get-zotero-data($url)
>     let $links := string($items[1]/descendant::*
> :header[@name='link']/@value)
>     let $last :=
>             for $last in tokenize($links,'&lt;')[contains(., ';
> rel="last"')]
>             return replace(substring-before($last,';
> rel="last"'),'&lt;|&gt;','')
>     let $next :=
>             for $next in tokenize($links,'&lt;')[contains(., ';
> rel="next"')]
>             return replace(substring-before($next,';
> rel="next"'),'&lt;|&gt;','')
>     return
>         <result>
>             <page url="{$url}"/>
>             { if($next) then local:process($next) else() }
>         </result>
> };
>
> <div>{
>     let $url := 'https://api.zotero.org/groups/538215/items?format=
> json&amp;limit=50'
>     return local:process($url)
>     }
> </div>
>
> It attempts to page through the Zotero collection. You should be able to
> run it from eXide, Zotero does not require any authentication to use the
> API.  The script is a simplified version of this:  https://github.com/bibl
> ia-arabica/zotero2bibl
>
> Thanks for looking into this.
> -Winona
>
>
>
>
>
> On Sun, Jan 14, 2018 at 7:25 AM, Adam Retter <ad...@ex...> wrote:
>
>> Hi Winona,
>>
>> I would definitely recommend sticking with the EXPath http client. I have
>> made some "improvements" to it in recent releases, where it now uses
>> connection pooling etc, it could be that one of these "improvements" is
>> causing you issue or has a problem. It would really help if I could debug
>> this on your systems.
>>
>> Is it possible for me to run your script here and access the Zotero
>> server?
>>
>> On 12 January 2018 at 14:34, Winona Salesky <wsa...@gm...> wrote:
>>
>>> Possibly related error message:
>>>
>>> 2018-01-12 09:26:50,663 [qtp1722326656-37] ERROR (StoredNode.java
>>> [getQName]:177) - Unknown node type: 4
>>>
>>> 2018-01-12 09:26:50,664 [qtp1722326656-37] ERROR (StoredNode.java
>>> [getQName]:177) - Unknown node type: 4
>>>
>>>
>>> This is bringing my development server to a halt. I really need some
>>> features of the eXistdb 3.6.x release, but it would be a lot of work to
>>> convert all of my  http:send-request() code to use the httpclient:get.
>>>
>>> Is anyone else experiencing issues with the http:send-request() in
>>> versions 3.6.x?
>>>
>>> Thanks,
>>> -Winona
>>>
>>> On Sun, Jan 7, 2018 at 9:32 AM, Winona Salesky <wsa...@gm...>
>>> wrote:
>>>
>>>> Thanks Dannes,
>>>> Sorry for not being more clear. The failure with the
>>>> http:send-request() happens most with repeated requests. So the submitted
>>>> code 'pages' through a Zotero collection so I can harvest all the items.
>>>> This script worked in version 3.3, but fails pretty consistently in 3.6.
>>>> When this code fails I never get any results returned, and the script does
>>>> not seem to time out or terminate.  The submitted code cycles through
>>>> Zotero requests until there is no 'next' page (parsed from the 'Link'
>>>> header).  It seems to be when I am making repeated requests that the http
>>>> client fails most often. The confusing part to me is that a single request
>>>> is usually fine, unless it follows a previous failure. After a failure I
>>>> have to restart the db to get any http:send-request() to work.
>>>>
>>>> If you run both submitted snippets you will see that the httpclient:get
>>>> is successful (although it can take a while), while the
>>>>  http:send-request() seems to fail pretty consistently in 3.6.0.
>>>>
>>>> Thanks for the help,
>>>> -Winona
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Jan 7, 2018 at 7:10 AM, Dannes Wessels <da...@ex...>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On 7 Jan 2018, at 4:32 , Winona Salesky <wsa...@gm...> wrote:
>>>>>
>>>>> Any ideas would be helpful.
>>>>>
>>>>>
>>>>> it would be more easy for the readers on exist-open to strip both
>>>>> code-snippets down to the minimum…..  in your case that is the output of
>>>>> the HTTP request.
>>>>>
>>>>> regards
>>>>>
>>>>> Dannes
>>>>>
>>>>>
>>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>>>
>>
>>
>> --
>> Adam Retter
>>
>> eXist Core Developer
>> { United Kingdom / United States }
>> ad...@ex...
>>
>
>