Penetration Testing: Perl and https requests

Thursday, 6 September 2012

Hi there,

today I was in the mood in writing some little perl script that I need for a project. To get the perl script running it was needed to execute some https requests.

First I was installing LWP::UserAgent and HTTP::Request via cpanm. Then I was writing a basis script that was executing a http request. I'm only interested in the header, so I don't want to print out the body content (I've found this litte code snippet here).

 #!/usr/bin/perl  
 use LWP::UserAgent; 
 use HTTP::Request;  

 my $URL = 'http://www.example.com/';  
 my $agent = LWP::UserAgent->new(env_proxy => 1,keep_alive => 1, timeout => 30);  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $agent->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 }

This worked for me very well, but I needed to create a https request. When I was executing the same script with https instead of http I was getting the following error:

 Error:https://www.example.com/  
 <html>  
 <head><title>An Error Occurred</title></head>  
 <body>  
 <h1>An Error Occurred</h1>  
 <p>501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)</p>  
 </body>  
 </html>

So, I need to install LWP:Protocol:https, but this wasn't working:

 $ sudo cpanm LWP::Protocol::https  
 --> Working on LWP::Protocol::https  
 Fetching http://www.cpan.org/authors/id/G/GA/GAAS/LWP-Protocol-https-6.03.tar.gz ... OK  
 Configuring LWP-Protocol-https-6.03 ... OK  
 ==> Found dependencies: IO::Socket::SSL  
 --> Working on IO::Socket::SSL  
 Fetching http://www.cpan.org/authors/id/S/SU/SULLR/IO-Socket-SSL-1.76.tar.gz ... OK  
 Configuring IO-Socket-SSL-1.76 ... OK  
 ==> Found dependencies: Net::SSLeay  
 --> Working on Net::SSLeay  
 Fetching http://www.cpan.org/authors/id/M/MI/MIKEM/Net-SSLeay-1.48.tar.gz ... OK  
 Configuring Net-SSLeay-1.48 ... OK  
 Building and testing Net-SSLeay-1.48 ... FAIL  
 ! Installing Net::SSLeay failed. See /root/.cpanm/build.log for details.  
 ! Bailing out the installation for IO-Socket-SSL-1.76. Retry with --prompt or --force.  
 ! Bailing out the installation for LWP-Protocol-https-6.03. Retry with --prompt or --force.

Openssl was installed, but I needed to install "build-essential libssl-dev" to get the installation of LWP:Protocol_https working:

 $ sudo apt-get install build-essential libssl-dev

Now https requests can be made with perl:

 #!/usr/bin/perl  
   
 use LWP::UserAgent;  
 use HTTP::Request;  
   
 my $URL = 'https://www.twitter.com/';  
   
 my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $ua->request($request);  
   
 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 }

Response of twitter.com on port 443:

   
 $ ./hsts.pl   
 URL:https://www.twitter.com/  
 Headers:  
 Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0  
 Date: Thu, 06 Sep 2012 21:22:33 GMT  
 Pragma: no-cache  
 ETag: "f7a8e95e2978ac6f73209336152b9495"  
 Server: tfe  
 Vary: Accept-Encoding  
 Content-Length: 47126  
 Content-Type: text/html; charset=utf-8  
 Expires: Tue, 31 Mar 1981 05:00:00 GMT  
 Last-Modified: Thu, 06 Sep 2012 21:22:33 GMT  
 Client-Date: Thu, 06 Sep 2012 21:22:33 GMT  
 Client-Peer: 199.59.148.10:80  
 Client-Response-Num: 1  
 Content-Base: http://twitter.com/  
 Link: <http://a0.twimg.com>; rel="dns-prefetch"  
 Link: <http://api.twitter.com>; rel="dns-prefetch"  
 Link: </favicons/favicon.ico>; rel="shortcut icon"; type="image/x-icon"  
 Link: <http://a0.twimg.com/a/1346884958/t1/css/t1_core_logged_out.bundle.css>; media="screen"; rel="stylesheet"; type="text/css"  
 Link: <https://twitter.com/>; rel="canonical"  
 Link: <http://a0.twimg.com/a/1346884958/t1/css/t1_more.bundle.css>; media="screen"; rel="stylesheet"; type="text/css"  
 Refresh: 0; URL=/?_twitter_noscript=1  
 Set-Cookie: k=10.36.21.101.1346966553057924; path=/; expires=Thu, 13-Sep-12 21:22:33 GMT; domain=.twitter.com  
 Set-Cookie: guest_id=v1%3A134696655306150737; domain=.twitter.com; path=/; expires=Sun, 07-Sep-2014 09:22:33 GMT  
 Set-Cookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCOaBdp05AToMY3NyZl9p%250AZCIlNThkNmZkYjY2ODJjNTc0MzY0YTY2Y2M0YjI0OGU2NWE6B2lkIiUwMmUz%250AZTJjY2VkMjFiYWNmZjQ5MmI2MjQyNWU5ZTJiMw%253D%253D--7d08430b6a85e0006ac4c062a4218d5cf841f564; domain=.twitter.com; path=/; HttpOnly  
 Status: 200 OK  
 Title: Twitter  
 X-Frame-Options: SAMEORIGIN  
 X-Meta-Charset: utf-8  
 X-Meta-Description: Verbinde Dich sofort mit den Dingen, die für Dich am wichtigsten sind. Folge Freunden, Experten, Lieblingsstars und aktuellen Nachrichten.  
 X-MID: e88c4d8fc53fc1466f24f3cbc905d24fd89af901  
 X-Runtime: 0.07026  
 X-Transaction: 4998cc5789e9b2c0  
 X-UA-Compatible: IE=edge  
 X-XSS-Protection: 1; mode=block

First step is done :-)

Perl and https requests

 #!/usr/bin/perl  
 use LWP::UserAgent; 
 use HTTP::Request;  

 my $URL = 'http://www.example.com/';  
 my $agent = LWP::UserAgent->new(env_proxy => 1,keep_alive => 1, timeout => 30);  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $agent->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 }