#24 htpps/ssl-requests not working over proxy

[Anonymous]

When trying to crawl a https/ssl-url over a defined proxy (setProxy), the proxy-server always responds "501 not implented".
This is due to the fact, that "CONNECT"-requests (instead og "GET"-requests) are not implemented yet.

[Anonymous]

I write some code in file libs/PHPCrawlerHTTPRequset.class.php that worked for me
I created function that building http header for ssl handshake

protected function buildProxySSLRequestHeader() {
// Create header
$headerlines = array();
// HTTP protocol
if ($this->http_protocol_version == PHPCrawlerHTTPProtocols::HTTP_1_1) $http_protocol_verison = "1.1";
else $http_protocol_verison = "1.0";

$headerlines[] = "CONNECT {$this->url_parts["host"]}:443 HTTP/{$http_protocol_verison}\r\n";
$headerlines[] = "Host: {$this->proxy['proxy_host']}\r\n";
$headerlines[] = "User-Agent: ".str_replace("\n", "", $this->userAgentString)."\r\n";
$headerlines[] = "Proxy-Connection: Keep-Alive\r\n";
$headerlines[] = "Connection: Keep-Alive\r\n";
$headerlines[] = "\r\n";

return $headerlines;
}

and in function sendRequest() after this lines

// If error occured
if ($PageInfo->error_code != null)
{
  // If proxy-error -> throw exception
  if ($PageInfo->error_code == PHPCrawlerRequestErrors::ERROR_PROXY_UNREACHABLE)
  {
    throw new Exception("Unable to connect to proxy '".$this->proxy["proxy_host"]."' on port '".$this->proxy["proxy_port"]."'");
  }

  $PageInfo->error_occured = true;
  return $PageInfo; 
}

paste this code

    // SSL handshake on proxy
if ($PageInfo->protocol == "https://" && $this->proxy != null) {
    $this->sendRequestHeader ($this->buildProxySSLRequestHeader ());

    $sslResponseHeader = new PHPCrawlerResponseHeader(
            $this->readResponseHeader($PageInfo->error_code, $PageInfo->error_string),
            $this->UrlDescriptor->url_rebuild
            );

    if ($sslResponseHeader->http_status_code != 200) {
        @fclose($this->socket);
        throw new Exception("Proxy SSL connection returned not 200 response");
    }

    $modes = array(
        STREAM_CRYPTO_METHOD_TLS_CLIENT,
        STREAM_CRYPTO_METHOD_SSLv3_CLIENT,
        STREAM_CRYPTO_METHOD_SSLv23_CLIENT,
        STREAM_CRYPTO_METHOD_SSLv2_CLIENT
    );

    $success = false;
    foreach($modes as $mode) {
        $success = stream_socket_enable_crypto($this->socket, true, $mode);
        if ($success) break;
    }
    if (!$success) {
        @fclose($this->socket);
        throw new Exception("Cannot secure connection");
    }
}

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

I forgot... also need to change one line in function buildRequestHeader() under comment "A Proxy needs the full qualified URL in the GET or POST headerline."

    if ($this->proxy != null)
    {
      // A Proxy needs the full qualified URL in the GET or POST headerline.
        $headerlines[] = $request_type." ".$this->url_parts["path"].$this->url_parts["file"].$this->url_parts["query"]." HTTP/{$http_protocol_verison}\r\n";
    }
    else
    {
      $query = $this->prepareHTTPRequestQuery($this->url_parts["path"].$this->url_parts["file"].$this->url_parts["query"]);
      $headerlines[] = $request_type." ".$query." HTTP/".$http_protocol_verison."\r\n";
    }

[Anonymous]

Hi!

What should i say, THANKS a lot for providing your code, great!

Will get implemented in the next release.

Thnks and cheers!

[Vinay]

I had issues getting crawling SSL enabled websites through a proxy. I implemented the code above and it works wonderfully well for most SSL enabled websites. However, for some websites, I get the following error

PHP Warning: stream_socket_enable_crypto(): SSL/TLS already set-up for this stream in /home/vinay/phpcrawl/libs/PHPCrawlerHTTPRequest.class.php on line 410
PHP Warning: stream_socket_enable_crypto(): SSL/TLS already set-up for this stream in /home/vinay/phpcrawl/libs/PHPCrawlerHTTPRequest.class.php on line 410
PHP Warning: stream_socket_enable_crypto(): SSL/TLS already set-up for this stream in /home/vinay/phpcrawl/libs/PHPCrawlerHTTPRequest.class.php on line 410
Cannot secure connection

The website that this can error can be replicated on is https://moz.com/

Any ideas on what I can do to fix it?

[Anonymous]

Same Problem as Vinay. Has anyone a solution?

[Anonymous]

IN PHPCrawlerHTTPRequest.class.php after
if ($this->proxy != null) {
I changed:

$context_proxy = stream_context_create(array('ssl' => array( 'verify_peer_name'=>false, 'SNI_enabled' => false)));
        $this->socket = stream_socket_client('tcp://'.$this->proxy["proxy_host"].":".$this->proxy["proxy_port"], $error_code, $error_str, $this->socketConnectTimeout, STREAM_CLIENT_CONNECT, $context_proxy);

Worked for me

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

[Anonymous]

As someone who just tried CBD for the at the outset stretch, I be required to bring up I'm genuinely impressed! I've been hearing about CBD like [url=https://greenrevolutioncbd.com/cbd-products/water-soluble-cbd/ ]water soluble cbd oil[/url] in search a while on occasion, but I was a bit skeptical almost how it would feign me. I decided to evaluate a small administer of CBD oil to get a load of if it would relief with my dyed in the wool shoulder distress and desire, and the results were more positive than I anticipated.

[Anonymous]

The article on the [url=https://dailycaller.com/2024/07/05/the-best-cbd-gummies-for-pain-in-2024/ ]Best CBD Gummies For Pain[/url] was incredibly neighbourly exchange for navigating the substantial array of products available. It provided full reviews and comparisons, highlighting factors like potency, ingredients, and customer feedback. The involvement of pros and cons as a service to each product was particularly beneficial in making informed choices. Whether you're a late drug or looking to switch brands, this article offers valuable insights to oversee your CBD/THC gummy selection.