Hello, first of all thanks to nice library. It is really helpful. But i am using jprofiler to analyze my app's memory management. Below you can see my webClient Options. I am using executorScheduledService and for every thread i am creating new Webclient because re-using same global webclient is a memory hog. Even with reinitiating it still uses memory a lot. The website i am logging and getting some information is bss.bimser.com.tr which is a small-moderate website. Some sample codes below;
WebClient webClient = new WebClient();
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(false);
webClient.getOptions().setAppletEnabled(false);
webClient.getOptions().setDownloadImages(false);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setHistorySizeLimit(0);
//webClient.getOptions().setMaxInMemory(10);
//webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getCookieManager().setCookiesEnabled(true);
webClient.waitForBackgroundJavaScript(1500);
webClient.setJavaScriptTimeout(1501);
if(loginCookies != null) {
webClient.setCookieManager(loginCookies);
}
Some usage
WebClient webClient = initWebClient();
try {
WebRequest webRequestPost = new WebRequest(new URL(DefaultValues.unassignedPostAddress), HttpMethod.POST);
webClient.getPage(webRequestPost);
WebRequest webRequestGet = new WebRequest(new URL(DefaultValues.unassignedGetAddress), HttpMethod.GET);
HtmlPage page = webClient.getPage(webRequestGet);
if (!page.asText().contains(DefaultValues.sessionOutKeyword) && page.asText().length() > 2) {
findUnassignedTickets(page);
}
else
needLogin = true;
}catch (Exception ex){
needLogin = true;
}
unassignedTickets.clear();
try {
List<HtmlTableRow> htrList = page.getByXPath(DefaultValues.ticketListKeyword);
for (HtmlTableRow row : htrList) {
int b = 0;
Ticket ticket = new Ticket();
for (HtmlTableCell cell : row.getCells()) {
mapUnassignedTicketFields(b, cell.asText(), ticket);
b++;
}
doUnassignedAfterMath(ticket);
}
}catch(IndexOutOfBoundsException ex){
ex.printStackTrace();
needLogin = true;
}
I tried this but no improvement either.
List<WebWindow> windows = webClient.getWebWindows();
for (WebWindow wd : windows) {
wd.getJobManager().removeAllJobs();
}
webClient.close();
jprofiler ss attached
Strage, you have javascript disabled and facing memory problems with javascript objects. If you can provide a simple, complete sample that is reproducible from here i will try to figure out what is going wrong.
If you do not like to post is here you can send it also to me via private mail (have a look at the team page).
you can see all code in the attach. Thanks
Had a quick look at your code. There are several places where you create new webClients. I like to suggest to really use the WebClient as an resource and only use it the safe way