today I was in the mood in writing some little perl script that I need for a project. To get the perl script running it was needed to execute some https requests.
First I was installing LWP::UserAgent and HTTP::Request via cpanm. Then I was writing a basis script that was executing a http request. I'm only interested in the header, so I don't want to print out the body content (I've found this litte code snippet here).
#!/usr/bin/perl
use LWP::UserAgent;
use HTTP::Request;
my $URL = 'http://www.example.com/';
my $agent = LWP::UserAgent->new(env_proxy => 1,keep_alive => 1, timeout => 30);
my $header = HTTP::Request->new(GET => $URL);
my $request = HTTP::Request->new('GET', $URL, $header);
my $response = $agent->request($request);
if ($response->is_success){
print "URL:$URL\nHeaders:\n";
print $response->headers_as_string;
}elsif ($response->is_error){
print "Error:$URL\n";
print $response->error_as_HTML;
}
This worked for me very well, but I needed to create a https request. When I was executing the same script with https instead of http I was getting the following error:
Error:https://www.example.com/
<html>
<head><title>An Error Occurred</title></head>
<body>
<h1>An Error Occurred</h1>
<p>501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)</p>
</body>
</html>
So, I need to install LWP:Protocol:https, but this wasn't working:
$ sudo cpanm LWP::Protocol::https
--> Working on LWP::Protocol::https
Fetching http://www.cpan.org/authors/id/G/GA/GAAS/LWP-Protocol-https-6.03.tar.gz ... OK
Configuring LWP-Protocol-https-6.03 ... OK
==> Found dependencies: IO::Socket::SSL
--> Working on IO::Socket::SSL
Fetching http://www.cpan.org/authors/id/S/SU/SULLR/IO-Socket-SSL-1.76.tar.gz ... OK
Configuring IO-Socket-SSL-1.76 ... OK
==> Found dependencies: Net::SSLeay
--> Working on Net::SSLeay
Fetching http://www.cpan.org/authors/id/M/MI/MIKEM/Net-SSLeay-1.48.tar.gz ... OK
Configuring Net-SSLeay-1.48 ... OK
Building and testing Net-SSLeay-1.48 ... FAIL
! Installing Net::SSLeay failed. See /root/.cpanm/build.log for details.
! Bailing out the installation for IO-Socket-SSL-1.76. Retry with --prompt or --force.
! Bailing out the installation for LWP-Protocol-https-6.03. Retry with --prompt or --force. Openssl was installed, but I needed to install "build-essential libssl-dev" to get the installation of LWP:Protocol_https working:
$ sudo apt-get install build-essential libssl-dev
Now https requests can be made with perl:
#!/usr/bin/perl
use LWP::UserAgent;
use HTTP::Request;
my $URL = 'https://www.twitter.com/';
my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
my $header = HTTP::Request->new(GET => $URL);
my $request = HTTP::Request->new('GET', $URL, $header);
my $response = $ua->request($request);
if ($response->is_success){
print "URL:$URL\nHeaders:\n";
print $response->headers_as_string;
}elsif ($response->is_error){
print "Error:$URL\n";
print $response->error_as_HTML;
}
Response of twitter.com on port 443:
$ ./hsts.pl
URL:https://www.twitter.com/
Headers:
Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
Date: Thu, 06 Sep 2012 21:22:33 GMT
Pragma: no-cache
ETag: "f7a8e95e2978ac6f73209336152b9495"
Server: tfe
Vary: Accept-Encoding
Content-Length: 47126
Content-Type: text/html; charset=utf-8
Expires: Tue, 31 Mar 1981 05:00:00 GMT
Last-Modified: Thu, 06 Sep 2012 21:22:33 GMT
Client-Date: Thu, 06 Sep 2012 21:22:33 GMT
Client-Peer: 199.59.148.10:80
Client-Response-Num: 1
Content-Base: http://twitter.com/
Link: <http://a0.twimg.com>; rel="dns-prefetch"
Link: <http://api.twitter.com>; rel="dns-prefetch"
Link: </favicons/favicon.ico>; rel="shortcut icon"; type="image/x-icon"
Link: <http://a0.twimg.com/a/1346884958/t1/css/t1_core_logged_out.bundle.css>; media="screen"; rel="stylesheet"; type="text/css"
Link: <https://twitter.com/>; rel="canonical"
Link: <http://a0.twimg.com/a/1346884958/t1/css/t1_more.bundle.css>; media="screen"; rel="stylesheet"; type="text/css"
Refresh: 0; URL=/?_twitter_noscript=1
Set-Cookie: k=10.36.21.101.1346966553057924; path=/; expires=Thu, 13-Sep-12 21:22:33 GMT; domain=.twitter.com
Set-Cookie: guest_id=v1%3A134696655306150737; domain=.twitter.com; path=/; expires=Sun, 07-Sep-2014 09:22:33 GMT
Set-Cookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCOaBdp05AToMY3NyZl9p%250AZCIlNThkNmZkYjY2ODJjNTc0MzY0YTY2Y2M0YjI0OGU2NWE6B2lkIiUwMmUz%250AZTJjY2VkMjFiYWNmZjQ5MmI2MjQyNWU5ZTJiMw%253D%253D--7d08430b6a85e0006ac4c062a4218d5cf841f564; domain=.twitter.com; path=/; HttpOnly
Status: 200 OK
Title: Twitter
X-Frame-Options: SAMEORIGIN
X-Meta-Charset: utf-8
X-Meta-Description: Verbinde Dich sofort mit den Dingen, die für Dich am wichtigsten sind. Folge Freunden, Experten, Lieblingsstars und aktuellen Nachrichten.
X-MID: e88c4d8fc53fc1466f24f3cbc905d24fd89af901
X-Runtime: 0.07026
X-Transaction: 4998cc5789e9b2c0
X-UA-Compatible: IE=edge
X-XSS-Protection: 1; mode=block
First step is done :-)
