#1948 Infinite scroll does not work (Google Photo Archive)

[Tamas Pflanzner]

I'm using these settings:

webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setCssEnabled(false);
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
webClient.getCurrentWindow().setInnerHeight(Integer.MAX_VALUE);
webClient.waitForBackgroundJavaScript(10000);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

If I try to crawl an album:

HtmlPage albumPage = webClient.getPage(PREFIX_GOOGLE_GET + albumRelLink)
// example: https://get.google.com/albumarchive/103435896752145457656/album/AF1QipM4quOBnp_cTvckNLl6u8QemRKcSITLH3FM4BVN
albumPage.getByXPath("//div[@ROLE='list']/div/img")

It only returns the first 24 items.

I think this Infinite scroll feature is usual on websites, so maybe worth supporting it.

[RBRi]

I think the aswer is yes and no :-). HtmlUnit, like a real browser loads only some inages when opening the page (have a look at the scroll bar in your browser). Seems like page scrolling triggers in some way the loading of additional content (javascript based).
It will be great if you can figure out what kind of event triggers this loading. Maybe there is a way to trigger this events from HtmlUnit also. I know there is a horrible amount of javascript in the page so it will be not that easy to understand the way it is implemented but it will be a great help if you can try to figure out how it works.

[RBRi]

Sorry but at the moment i have no idea hwo to fix this. Will be great if you can dig into the js and try to provide some more insights how this works in real browsers.

[Davit Mkrtchyan]

Hi all
I think I got some solution, it is just run js like this

var scrollSize = 200;
var period = 300;
var html = false;

function doPeriod() {
return new Promise(resolve => {
setTimeout(() => {
var prev = window.scrollY;
window.scroll(0, window.scrollY + scrollSize);
if (window.scrollY > prev) {
resolve(false);
} else {
resolve(document.body.innerHTML);
}
}, period);
});
}

async function scrolling() {
while (html === false) {
html = await doPeriod();
}
return html;
}

var x = await scrolling();
return x;

but the problem is that I'm getting exception
com.gargoylesoftware.htmlunit.ScriptException: missing ; before statement (injected script#19)

it is because the framework dows not support javascript async functions.
Any other ideas to fix it?