Re: [Htmlunit-user] Help Extracting Schedule from a Website

[Rich G. <ri...@um...>]

Ah, this was an issue with the website.

I had to load the page:
http://alisondb.legislature.state.al.us/Alison/SESSBillsList.aspx?STATUSCODES=Had%20First%20Reading%20House%20of%20Origin&BODY=1755
and then load the provided link.

Thanks for thinking about this.
    -Rich

On Wed, Aug 12, 2015 at 6:57 PM, Rich Goldman <ri...@um...> wrote:

> Right, that is static text. It doesn't include the legislation title,
> related documents, legislative actions, and other information that loads
> when using a web browser.
>
>
> On Wednesday, August 12, 2015, Ahmed Ashour <asa...@ya...> wrote:
>
>> Hi Rich,
>>
>> Please be more specific.
>>
>> Using latest snapshot, INTERNET_EXPLORER_8, and .asText(), I get the
>> below, which is similar to what is in the website.
>>
>>
>> Alabama Legislature
>> First Special Session 2015
>>
>>
>>
>>    Home
>> Session Information
>> Code of Alabama
>> Constitution
>> House of Representatives
>> Senate
>> Help
>> Contact Us
>> Legislative Day
>> Console
>> Bills
>> Resolutions
>> Confirmations
>> Committee Meetings
>> Interim Meetings &
>>  Announcements
>> Reports
>> Adjourn Convene
>> Legislative Audio/Video
>> Bill Status for HB1 ()
>> Not Official Information from the Legislative Offices
>> Document Quick Links
>> Legislative Process
>> Legislative Glossary
>> Alabama Legislature
>> The Legislature convenes in regular annual sessions on the first Tuesday
>> in February, except (1) in the first year of the four-year term, when the
>> session will begin on the first Tuesday in March, and (2) in the last year
>> of a four-year term, when the session will begin on the second Tuesday in
>> January. The length of the regular session is limited to 30 meeting days
>> within a period of 105 calendar days. There are usually two meeting or
>> "legislative" days per week, with other days devoted to committee meetings.
>> Special sessions of the Legislature may be called by the Governor, with the
>> Proclamation listing the subjects which the Governor wishes considered.
>> These sessions are limited to 12 legislative days within a 30 calendar day
>> span. In a regular session, bills may be enacted on any subject. In a
>> special session, legislation must be enacted only on those subjects which
>> the Governor announces in his proclamation or "call." Anything not in the
>> "call" requires a two-thirds vote of each house to be enacted.
>> Alabama Legislature
>>
>>
>> ------------------------------
>> *From:* Rich Goldman <ri...@um...>
>> *To:* Ahmed Ashour <asa...@ya...>; "
>> htm...@li..." <htm...@li...>
>>
>> *Sent:* Thursday, August 13, 2015 12:30 AM
>> *Subject:* Re: [Htmlunit-user] Help Extracting Schedule from a Website
>>
>>
>>
>> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible
>> for Automatic Cleanup! (ri...@um...) Add cleanup rule
>> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DAMgvRVPUwyF2iRLLTI0umlXbZ6SAWgjct8HRmSQZ9JdmzVFeVOAZ4HzjNoYTF6bGFVyvLcO1NYQmjge%252BLLJemjtvWsxiczf3dzdjoZQBcA1mX4T4GWxwO5BOcU2reMr9XexwsK956GY%253D%26key%3Dv2%252Bzz5t79i%252FU%252F0TwW0y%252Fn060KVhYo7FuIAD%252BeRhm0Ng%253D&tc_serial=22274169521&tc_rand=840714072&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>> | More info
>> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=22274169521&tc_rand=840714072&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>>
>> If you go to the webpage in a browser you'll get all sorts of information
>> on a piece of legislation.  This code simply returns the template without
>> any data/html populated.  Does this make sense?
>>
>> On Wednesday, August 12, 2015, Ahmed Ashour <asa...@ya...> wrote:
>>
>> Hi,
>>
>> You can also use Thread.sleep().
>>
>> What is missing (not loaded)?
>>
>> Please read http://htmlunit.sourceforge.net/submittingJSBugs.html
>>
>> Ahmed
>>
>> ------------------------------
>> *From:* Rich Goldman <ri...@um...>
>> *To:* htm...@li...
>> *Sent:* Wednesday, August 12, 2015 10:22 PM
>> *Subject:* Re: [Htmlunit-user] Help Extracting Schedule from a Website
>>
>>
>>
>> I am trying to get the populated HTML of another site but it is not
>> loading, despite putting the wait time to 30 seconds.
>>
>> The address is
>> http://alisondb.legislature.state.al.us/Alison/SESSBillResult.aspx?BILL=HB1&WIN_TYPE=SELECTED_STATUS
>>
>>
>> Are you all able to get page.asXml();  to produce populated html for
>> this address?
>>
>> I've updated to 2.18 and I've tried putting
>> the waitForBackgroundJavaScript in multiple places without success. My code
>> is:
>>
>> public String getWebsiteTextWithJavaScript(String url) {
>> WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_6);
>> HtmlPage page = null;
>> try {
>> webClient.waitForBackgroundJavaScript(30000);
>> page = webClient.getPage(url);
>> } catch (FailingHttpStatusCodeException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (MalformedURLException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>> //
>> // // Thread.sleep(10000);
>> webClient.waitForBackgroundJavaScript(30000);
>> String text = page.asXml();
>> webClient.waitForBackgroundJavaScript(30000);
>> page.cleanUp();
>> webClient.closeAllWindows();
>>
>> return text;
>> }
>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Htmlunit-user mailing list
>> Htm...@li...
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>>
>>
>>