Menu
▾
▴
Re: [Htmlunit-user] Help Extracting Schedule from a Website
|
From: Ahmed A. <asa...@ya...> - 2015-03-04 16:45:46
|
Hi Rich, Well, waitForBackground is actually better than Thread.sleep(), and it works. It didn't work with you before, because you used it 'before' webClient.getPage(), however it should be 'after', to allow JavaScript/AJAX to run. Hope that clarifies, Ahmed From: Rich Goldman <ri...@um...> To: htm...@li... Sent: Wednesday, March 4, 2015 4:56 PM Subject: Re: [Htmlunit-user] Help Extracting Schedule from a Website I think I was confused between using Thread.sleep(10000) and webClient.waitForBackgroundJavaScript(10000). Thanks again. -Rich On Wed, Mar 4, 2015 at 10:21 AM, Alain BUFERNE <alb...@gm...> wrote: By using HtmlUnit, you generally just program what a normal human being will do to use the webSite. Since you just need information send by the server in response of clickt this, select that, you don't need to execute Js code . 2015-03-04 7:05 GMT+01:00 Rich Goldman <ri...@um...>: Doing a bit more digging, it seems the javascript functions for populating the agenda items are in: http://www.house.leg.state.mn.us/schedules/ScheduleElements0.js?v=1.12 I don't know enough javascript to know how to execute these functions appropriately though. -Rich On Wed, Mar 4, 2015 at 12:41 AM, Rich Goldman <ri...@um...> wrote: I'm trying to get the schedule information posted at: http://www.house.leg.state.mn.us/schedules/schedule.aspx#03/06/2015 The content is loaded dynamically (presumably via AJAX) and I've tried the following code: final WebClient webClient = new WebClient(BrowserVersion.CHROME); webClient.waitForBackgroundJavaScript(10000); final HtmlPage page = webClient .getPage("http://www.house.leg.state.mn.us/schedules/schedule.aspx#03/06/2015"); String javaScriptCode = "SchedJSx.Init();"; ScriptResult result = page.executeJavaScript(javaScriptCode); result.getJavaScriptResult(); System.out.println("result: " + result.getJavaScriptResult()); I can get some of the dynamic content:Friday, March 06, 201510:30 AMHealth and Human Services ReformChair: Rep. Tara MackLocation: Basement State Office BuildingNote:***Additional bills may be added but not the agenda/bill list. I feel like I'm missing something simple that I'm now aware of as a newbie. I would appreciate a skilled HTML Unit user looking at the source code of the source website and pointing out what I'm missing so I can extract the agenda for this meeting as well. Thanks for any help you can provide. -Rich ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Htmlunit-user mailing list Htm...@li... https://lists.sourceforge.net/lists/listinfo/htmlunit-user ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Htmlunit-user mailing list Htm...@li... https://lists.sourceforge.net/lists/listinfo/htmlunit-user ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ Htmlunit-user mailing list Htm...@li... https://lists.sourceforge.net/lists/listinfo/htmlunit-user |
