Https not supported

[Dan]

HTTPS is not supported?

Exception in thread "main" org.webharvest.exception.HttpException: IO error during HTTP execution for URL:
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:80)
at org.webharvest.runtime.processors.HttpProcessor.execute(HttpProcessor.java:46)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.HtmlToXmlProcessor.execute(HtmlToXmlProcessor.java:27)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:34)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyListContent(BaseProcessor.java:104)
at org.webharvest.runtime.processors.LoopProcessor.execute(LoopProcessor.java:39)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.FileProcessor.executeFileWrite(FileProcessor.java:67)
at org.webharvest.runtime.processors.FileProcessor.execute(FileProcessor.java:40)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.Scraper.execute(Scraper.java:73)
at org.webharvest.runtime.Scraper.execute(Scraper.java:82)
at com.abc.WebHarvestTest.main(WebHarvestTest.java:25)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:825)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:393)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:168)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:58)
... 22 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at sun.security.validator.Validator.validate(Unknown Source)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(Unknown Source)
... 40 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.security.cert.CertPathBuilder.build(Unknown Source)
... 45 more

[Dan]

Great! I'll try it tomorrow.
Thanks.

[Vladimir Nikic]

It should be supported, but what is the URL you are trying to access?

[Dan]

i've noticed it doesn't support any self-signed web site. here is the url: https://www.consulfrance-londres.org/consulat/rdv/visa/etranger/calendrier.php

since you are using commons-httpclient library, i guess you need to handle self-signed website different. EasySSLProtocolSocketFactory and EasyX509TrustManager provided by the sample code is a good example.

[Vladimir Nikic]

Thanks for your update. Such user response is really precious - it is the best possible way to find bugs and weaknesses.
I'll check this as soon as possible.

[Vladimir Nikic]

Support for HTTPS pages with self-signed certificates added.
Please download new version (0.23).