Exception in thread "main" org.webharvest.exception.HttpException: IO error during HTTP execution for URL:
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:80)
at org.webharvest.runtime.processors.HttpProcessor.execute(HttpProcessor.java:46)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.HtmlToXmlProcessor.execute(HtmlToXmlProcessor.java:27)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:34)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyListContent(BaseProcessor.java:104)
at org.webharvest.runtime.processors.LoopProcessor.execute(LoopProcessor.java:39)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.FileProcessor.executeFileWrite(FileProcessor.java:67)
at org.webharvest.runtime.processors.FileProcessor.execute(FileProcessor.java:40)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.Scraper.execute(Scraper.java:73)
at org.webharvest.runtime.Scraper.execute(Scraper.java:82)
at com.abc.WebHarvestTest.main(WebHarvestTest.java:25)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:825)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:393)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:168)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:58)
... 22 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at sun.security.validator.Validator.validate(Unknown Source)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(Unknown Source)
... 40 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.security.cert.CertPathBuilder.build(Unknown Source)
... 45 more
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
since you are using commons-httpclient library, i guess you need to handle self-signed website different. EasySSLProtocolSocketFactory and EasyX509TrustManager provided by the sample code is a good example.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for your update. Such user response is really precious - it is the best possible way to find bugs and weaknesses.
I'll check this as soon as possible.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
HTTPS is not supported?
Exception in thread "main" org.webharvest.exception.HttpException: IO error during HTTP execution for URL:
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:80)
at org.webharvest.runtime.processors.HttpProcessor.execute(HttpProcessor.java:46)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.HtmlToXmlProcessor.execute(HtmlToXmlProcessor.java:27)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:34)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyListContent(BaseProcessor.java:104)
at org.webharvest.runtime.processors.LoopProcessor.execute(LoopProcessor.java:39)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.processors.BaseProcessor.executeBody(BaseProcessor.java:72)
at org.webharvest.runtime.processors.BaseProcessor.getBodyTextContent(BaseProcessor.java:85)
at org.webharvest.runtime.processors.FileProcessor.executeFileWrite(FileProcessor.java:67)
at org.webharvest.runtime.processors.FileProcessor.execute(FileProcessor.java:40)
at org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:51)
at org.webharvest.runtime.Scraper.execute(Scraper.java:73)
at org.webharvest.runtime.Scraper.execute(Scraper.java:82)
at com.abc.WebHarvestTest.main(WebHarvestTest.java:25)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:825)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:393)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:168)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at org.webharvest.runtime.web.HttpClientManager.execute(HttpClientManager.java:58)
... 22 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at sun.security.validator.Validator.validate(Unknown Source)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(Unknown Source)
... 40 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.security.cert.CertPathBuilder.build(Unknown Source)
... 45 more
Great! I'll try it tomorrow.
Thanks.
It should be supported, but what is the URL you are trying to access?
i've noticed it doesn't support any self-signed web site. here is the url: https://www.consulfrance-londres.org/consulat/rdv/visa/etranger/calendrier.php
since you are using commons-httpclient library, i guess you need to handle self-signed website different. EasySSLProtocolSocketFactory and EasyX509TrustManager provided by the sample code is a good example.
Thanks for your update. Such user response is really precious - it is the best possible way to find bugs and weaknesses.
I'll check this as soon as possible.
Support for HTTPS pages with self-signed certificates added.
Please download new version (0.23).