Re: [Htmlunit-user] Help Extracting Schedule from a Website

[Rich G. <ri...@um...>]

Right, that is static text. It doesn't include the legislation title,
related documents, legislative actions, and other information that loads
when using a web browser.

On Wednesday, August 12, 2015, Ahmed Ashour <asa...@ya...> wrote:

> Hi Rich,
>
> Please be more specific.
>
> Using latest snapshot, INTERNET_EXPLORER_8, and .asText(), I get the
> below, which is similar to what is in the website.
>
>
> Alabama Legislature
> First Special Session 2015
>
>
>
>    Home
> Session Information
> Code of Alabama
> Constitution
> House of Representatives
> Senate
> Help
> Contact Us
> Legislative Day
> Console
> Bills
> Resolutions
> Confirmations
> Committee Meetings
> Interim Meetings &
>  Announcements
> Reports
> Adjourn Convene
> Legislative Audio/Video
> Bill Status for HB1 ()
> Not Official Information from the Legislative Offices
> Document Quick Links
> Legislative Process
> Legislative Glossary
> Alabama Legislature
> The Legislature convenes in regular annual sessions on the first Tuesday
> in February, except (1) in the first year of the four-year term, when the
> session will begin on the first Tuesday in March, and (2) in the last year
> of a four-year term, when the session will begin on the second Tuesday in
> January. The length of the regular session is limited to 30 meeting days
> within a period of 105 calendar days. There are usually two meeting or
> "legislative" days per week, with other days devoted to committee meetings.
> Special sessions of the Legislature may be called by the Governor, with the
> Proclamation listing the subjects which the Governor wishes considered.
> These sessions are limited to 12 legislative days within a 30 calendar day
> span. In a regular session, bills may be enacted on any subject. In a
> special session, legislation must be enacted only on those subjects which
> the Governor announces in his proclamation or "call." Anything not in the
> "call" requires a two-thirds vote of each house to be enacted.
> Alabama Legislature
>
>
> ------------------------------
> *From:* Rich Goldman <ri...@um...
> <javascript:_e(%7B%7D,'cvml','ri...@um...');>>
> *To:* Ahmed Ashour <asa...@ya...
> <javascript:_e(%7B%7D,'cvml','asa...@ya...');>>; "
> htm...@li...
> <javascript:_e(%7B%7D,'cvml','htm...@li...');>" <
> htm...@li...
> <javascript:_e(%7B%7D,'cvml','htm...@li...');>>
> *Sent:* Thursday, August 13, 2015 12:30 AM
> *Subject:* Re: [Htmlunit-user] Help Extracting Schedule from a Website
>
>
>
> [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible
> for Automatic Cleanup! (ri...@um...
> <javascript:_e(%7B%7D,'cvml','ri...@um...');>) Add cleanup rule
> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DAMgvRVPUwyF2iRLLTI0umlXbZ6SAWgjct8HRmSQZ9JdmzVFeVOAZ4HzjNoYTF6bGFVyvLcO1NYQmjge%252BLLJemjtvWsxiczf3dzdjoZQBcA1mX4T4GWxwO5BOcU2reMr9XexwsK956GY%253D%26key%3Dv2%252Bzz5t79i%252FU%252F0TwW0y%252Fn060KVhYo7FuIAD%252BeRhm0Ng%253D&tc_serial=22274169521&tc_rand=840714072&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
> | More info
> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=22274169521&tc_rand=840714072&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>
> If you go to the webpage in a browser you'll get all sorts of information
> on a piece of legislation.  This code simply returns the template without
> any data/html populated.  Does this make sense?
>
> On Wednesday, August 12, 2015, Ahmed Ashour <asa...@ya...
> <javascript:_e(%7B%7D,'cvml','asa...@ya...');>> wrote:
>
> Hi,
>
> You can also use Thread.sleep().
>
> What is missing (not loaded)?
>
> Please read http://htmlunit.sourceforge.net/submittingJSBugs.html
>
> Ahmed
>
> ------------------------------
> *From:* Rich Goldman <ri...@um...>
> *To:* htm...@li...
> *Sent:* Wednesday, August 12, 2015 10:22 PM
> *Subject:* Re: [Htmlunit-user] Help Extracting Schedule from a Website
>
>
>
> I am trying to get the populated HTML of another site but it is not
> loading, despite putting the wait time to 30 seconds.
>
> The address is
> http://alisondb.legislature.state.al.us/Alison/SESSBillResult.aspx?BILL=HB1&WIN_TYPE=SELECTED_STATUS
>
>
> Are you all able to get page.asXml();  to produce populated html for this
> address?
>
> I've updated to 2.18 and I've tried putting
> the waitForBackgroundJavaScript in multiple places without success. My code
> is:
>
> public String getWebsiteTextWithJavaScript(String url) {
> WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_6);
> HtmlPage page = null;
> try {
> webClient.waitForBackgroundJavaScript(30000);
> page = webClient.getPage(url);
> } catch (FailingHttpStatusCodeException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (MalformedURLException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> //
> // // Thread.sleep(10000);
> webClient.waitForBackgroundJavaScript(30000);
> String text = page.asXml();
> webClient.waitForBackgroundJavaScript(30000);
> page.cleanUp();
> webClient.closeAllWindows();
>
> return text;
> }
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Htmlunit-user mailing list
> Htm...@li...
> <javascript:_e(%7B%7D,'cvml','Htm...@li...');>
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>
>
>