Squid 3.3.10 - Transparent Proxy for HTTP and HTTPS

Hey there,

for several years the squid proxy can be used as transparent proxy for HTTP and also HTTPS. As I was curious how it will work and how hard it is to setup, I've just installed and configured it.

First I installed a fresh virtual machine with Debian 7.2. In Debian you could either install Squid 2.7 or Squid 3.1 via apt-get (apt-get install squid or apt-get install squid3). Unfortunately to make a transparent proxy that also supports all HTTPS features, at least version 3.2 is needed. So I downloaded the latest sources (Version 3.3.10) directly from squid-cache.org. Before installing, the following packages should be installed in Debian, otherwise errors will pop-up during configure or make:

# apt-get install build-essential
# apt-get install libssl-dev

After unpacking the squid sources it is important to use the following configure statement, to activate ssl, because it is disabled by default:

#./configure --prefix=/usr/local/squid --enable-icap-client --enable-ssl --enable-ssl-crtd --with-default-user=squid

Afterwards you can compile and install squid:

# make all
# sudo make install

Now squid is installed in /usr/local/squid. As next step the user squid should be created and the log directory should be allocated to that user:

# useradd squid
# chown -R squid:squid /usr/local/squid/var/logs/

The next steps I've copied from the squid documentation (2): 

Afterwards you must create the swap directories. Do this by running Squid with the -z option:

# /usr/local/squid/sbin/squid -z

Once the creation of the cache directories completes, you can start Squid and try it out. Probably the best thing to do is run it from your terminal and watch the debugging output. Use this command:

# /usr/local/squid/sbin/squid -NCd1

If everything is working okay, you will see the line:

Ready to serve requests. 

If you want to run squid in the background, as a daemon process, just leave off all options:

# /usr/local/squid/sbin/squid

Now you should have a running squid on port 3128. But we still do not support HTTPS requests and the proxy is still not transparent. The next steps will be modifing squid.conf and put in some iptables rules. But at first we need to create our your own CA (Certificate Authority):

# cd /usr/local/squid
# mkdir ssl_cert
# cd ssl_cert
# openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout myCA.pem -out myCA.pem

This pem file can now be imported in your certificate store in your browser. Then you will not get any certificate errors when surfing HTTPS sites later via our transparent squid.
Next we need to replace the line "http_port 3128" with the following lines in /usr/local/squid/etc/squid.conf:

http_port 3128 intercept
https_port 3127 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/usr/local/squid/ssl_cert/myCA.pem
acl broken_sites dstdomain .example.com
ssl_bump none localhost
ssl_bump none broken_sites
ssl_bump server-first all
sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s /usr/local/squid/var/lib/ssl_db -M 4MB
sslcrtd_children 5

Also ip-forwarding needs to be activated:

# echo "1" > /proc/sys/net/ipv4/ip_forward

Finaly we need to insert our iptables rules to redirect the traffic to squid:

# iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
# iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 3127
# iptables -I INPUT -p tcp -m tcp --dport 3127 -j ACCEPT

Another folder needs to be created, for the dynamically generated certificates:

# mkdir /usr/local/squid/var/lib
# /usr/local/squid/libexec/ssl_crtd -c -s /usr/local/squid/var/lib/ssl_db -M 4MB
# chown -R squid:squid /usr/local/squid/var/lib/ssl_db/

Now you should start squid in debugging mode:

# /usr/local/squid/sbin/squid -NCd9

If the process is running and you get something similar to this, you work was successfull:

2013/11/04 22:39:16| Accepting NAT intercepted HTTP Socket connections at local=0.0.0.0:3128 remote=[::] FD 19 flags=41
2013/11/04 22:39:16| Accepting NAT intercepted SSL bumped HTTPS Socket connections at local=0.0.0.0:3127 remote=[::] FD 20 flags=41

To fully work as transparent HTTPS proxy, your clients in the network needs now the IP of this proxy as gateway address and the pem certificate needs to be imported in the browser of the clients. 

Now you can start squid by exeuting:

# /usr/local/squid/sbin/squid

Debugging:

If you have any problems you should check if squid and their ports are running. You can do this by using netstat:

# netstat -tulpen

You should then see port 3128 and 3127. If not execute "killall squid" several times and restart squid in debugging mode with 

# /usr/local/squid/sbin/squid -NCd9

You can also have a look at the access.log during browsing or via tcpdump to see if the packets are really arriving at your proxy. 

Hint:

This was just a quick'n'dirty how-to on how a transparent proxy supporting HTTPS can be created. This setup is for lab environments to get to know squid and it's capabilites and not for productive use. For example your private key is in the pem certificate which should be seperated from the certificate your deploying to your browser. 

Links
(1) Download Squid Source
(2) Installing Squid
(3) Dynamics SSL Certificate Generation
(4) SSL Bump

X Forwarding with SSH - Magic-Cookie problem

Hey there,

I've mentioned in one of my last posts, that it is possible to forward X via SSH. In my case I'm connecting from my Mac OS X client to my Rasperry PI running Kali Linux. I'm using the X forwarding feature of ssh to start tools that would need X on my Raspberry Pi, but the window will pop up in Mac OS X, as long as X11 is started on my Mac. If this was too confusing, you can just read this, and I think you will get it ;-)

I've just got one problem when doing this: When I log into Kali Linux I'm using an unprivileged account, let's say the account name is alice. The problem is that some tools need root-privileges, like Wireshark (of course you can also run tcpdump, but Wireshark is just an example). If I switch to the root account via su, the X forwarding for the application I want to start is not working anymore:

root@kali:~# wireshark
(wireshark:2810): Gtk-WARNING **: cannot open display: localhost:11.0

I'm getting this error because when the ssh connection is initiated a file called .Xauthority is created in the home directory of alice. This file contains a "session cookie" called Magic-Cookie. When I want to start now the application as root, the content of this file is not available to the root account, so I have to copy the .Xauthority file to the home folder of the root account:

# su -
# cp /home/alice/.Xauthoriy /root/

Then the Magic-Cookie will also be available for the root account and now wireshark can be started. If it is still not working you should check the environment variable DISPLAY. The DISPLAY variable of alice needs to be the same as in the root account.

To automate this task, I've created the file .bash_profile in the root directory:

# touch /root/.bash_profile
# vim /root/.bash_profile

and added the following content:

# cp /home/alice/.Xauthoriy /root/

Now everytime when I change to the root account the .Xauthority will be copied in the home folder of the root account and the X forwaring feature is still working.

If you have better/other solutions for this problem, feel free to leave a comment.
Cheers.

ssh and tmux

Hi there,

with tmux you can make your life a little more easy, if you have to work on the command line or manage one or more servers. So here is what I did:

If I connect to one of my servers via ssh I'm doing this always via my ssh key. Here you can find a detailed guide on how to setup a connection via ssh by using a key and a password for the key. If you're using this kind of authentication you just have to remember one password (the one for your private key) and you can login to any server you distributed your public key to. So now you can easily connect to your server(s) without creating another password for your user on another server.

But after login you have still just one shell available but maybe you need sometimes more shells but don't want to login for it. Thats the moment when you should start tmux.

As there are already some good tutorials and explanations I don't want to make my own one here so just visit here, here, here (german) or here as a starting point.

There is also a book available about tmux. I didn't read it, but maybe useful for someone who wants to dive deeper into tmux.

Cheers.

Be your own cloud provider and kick out Google Calendar, Dropbox and co. - Part 2 File Sync

Hey there,

so after I was able to sync all my calendar entries with all my devices by using OwnCloud, the next step is to use it as Dropbox replacement.

The main purpose for me using Dropbox was always to store documents like PDFs (books, whitepapers etc.) and to read them on my iPad. It was very convenient  and I didn't had to worry about backups, as the files were on my mobile devices, my laptop and in my Dropbox and it was also very convenient to share the files with others.

Here is the configuration, that I'm using now instead of Dropbox:

1. iOS: I'm using an app called "Good Reader" on my iPad in order to read all kinds of documents and Good Reader also provides an interface to connect to your Dropbox. It is possible for every App that can talk to a WebDAV server to connect to your OwnCloud. In Good Reader I just needed to add a new WebDAV server,  insert the URL accordingly to the manual of OwnCloud (e.g. https://example.com/owncloud/) put in your credentials and afterwards you can sync all data with Good Reader. You can sync data that is already available in your OwnCloud with Good Reader or upload files to OwnCloud via Good Reader. It's working for me now as convenient as Dropbox. 
2. Mac OS X: I wanted to use the Finder of Mac OS X for connecting to OwnCloud, as described here. Unfortunately it is not working as expected. I'm able to connect to my server via WebDAV and I can navigate through the directories, but when I want to create a folder or upload a file, it takes minutes and then the operation I wanted to execute did not succeed. I couldn't find out the problem, so I switched to Cyberduck. With Cyberduck I'm not having any problems and it I've got a good performance. 
3. Windows: In Windows it was no problem to map the WebDAV share to a drive letter. Maybe you need to tweak on the registry, but I didn't had to do it on my Windows 7 Professional Laptop. 

With this configuration I can access now all my files via iOS, Mac OS X and Windows. But to access the files I need to be online, otherwise the files will not be available. To access your files also when your offline you can use the Sync-Clients by OwnCloud.
In Windows it worked without any errors, but on Mac OS X I alway got the following error when I wanted to connect to my server via HTTPS:

Die Verbindung zu OwnCloud konnte nicht hergestellt werden: Im Ablauf des SSL-Protokolls ist ein Fehler aufgetreten.

respectively in english:

Failed to connect to ownCloud: SSL-Handshake failed

When I added the parameter "ServerName" to /etc/apache2/apache2.conf and did a restart of apache I was also able to connect to my OwnCloud with the Mac OS X Sync-Client.

So now I can share my calendar and files between all my devices with my OwnCloud and do not have to use Dropbox, iCloud or another cloud provider.

Great success :-)

Be your own cloud provider and kick out Google Calendar, Dropbox and co. - Part 1 Calendar

Hey there,

I want to make a little experiment to get as much data in my own cloud and not using services like Google Calendar or iCloud. Especially because of all the things regarding Edward Snowden disclosed confirmed all our paranoid thoughts about a big brother scenario and total surveillance and I want to try to be the master of my data as much as possible now. And of course I'm curious what can be done with services like OwnCloud.

What I want is to be my own cloud provider by using my own root-server and all my devices (laptop, smartphone and tablet) can use this server to sync their data. Actually I've used such cloud services like Google Calendar or Dropbox, but never trusted those services and I always felt uncomfortable and thats why I  didn't use cloud services e.g. for syncing my contacts. I've always synced my contacts directly via my laptop to my other devices.

My goal is to get as much data on my own server so it's under my control and not stored on some server or on a server somewhere in the US that will be monitored by some agency. Of course this server will be the single point of failure, and if it get's hacked all my data will be disclosed or compromised, but hey, at least I'm responsible now for my data.

First thing I've done is installing OwnCloud on my Debian server, see the link here for further installation instructions. Afterwards you can navigate to your web server by adding /owncloud to your URL, e.g. https://www.dummy.org/owncloud for further configuration.

I wanted to use the MySQL service as database for OwnCloud, as it is already running on my server:

1. Connect to MySQL and create a database for OwnCloud:

root@kali # mysql -u root -p
mysql> create database owncloud;

2. Create a user for the new database owncloud and grant all privileges to him

mysql> GRANT ALL PRIVILEGES
    -> ON owncloud.*
    -> TO 'owncloudUser'@'localhost'
    -> IDENTIFIED BY '<your password here>'
    -> WITH GRANT OPTION;

3. Now you can go again to https://www.dummy.org/owncloud and type in the name of the database you want to use for OwnCloud and the user and password for it. Also an administrator user will be created for the web interface.

Afterwards you can finish the installation and your OwnCloud is ready. You should also use SSL for your OwnCloud, so that your communication channel is encrypted. If you don't use SSL now and you don't want to spend money for an SSL certificate you should consider to create a server certificate at CAcert. Don't forget to import the root Certificate of CAcert into your browser and devices that want to use OwnCloud, so you have a trusted connection to your server.

So what can you do now by using OwnCloud? After logging into your OwnCloud you have the opportunity to share a calendar, contacts, data, pictures and music.

I just wanted to use the calendar service for now. To synchronize the calendar with your iOS device, just follow the manual at owncloud.org. You can also synchronize it with iCal on OS X and also with Lightning in Thunderbird. In Lightning you need the CalDav link that points directly to your calendar. You can find that link in OwnCloud 5 if you navigate to calendar, click on the settings symbol in the right corner and click on the little earth symbol in the row of your calendar. Then the CalDav link will appear. In Lightning you just need to create a new calendar, choose network, select CalDav as format and paste the URL in the address field. Then you just need to fill in the credentials in the login dialog that will pop-up and you have also the OwnCloud calendar in Thunderbird Lightning.

You can also sync the calendar with Android devices, but you need a 3rd party app like Card-Dav Sync. As I've got no Android device, I could not test it, so if there are better apps or if it is supported by the OS by now, feel free to leave a comment.

For me this setup is working fine now. I'm using it on OS X in iCal, on my iPhone, iPad and also another Windows Laptop has access to the calendar via Thunderbird Lightning and all via SSL. First step is done to get your own secure datastore.

For backup purposes here is a little hint what you want to backup on another machine to restore your data in OwnCloud, if the server crashes.

Cheers.

Raspberry Pi and Nano USB WiFi (EDIMAX EW-7811Un) on Kali Linux

Hey there,

yesterday my order arrived. An Wireless USB Adapter for my Raspberry Pi. Right after plugging it into the Pi and booting it up, it was found:

root@kali:~# dmesg

...
usb 1-1.2: new high-speed USB device number 4 using dwc_otg
usb 1-1.2: New USB device found, idVendor=7392, idProduct=7811
usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.2: Product: 802.11n WLAN Adapter
usb 1-1.2: Manufacturer: Realtek
usb 1-1.2: SerialNumber: 00e04c000001

You can also list the usb devices via lsusb, to make sure the device is recognized:

root@kali:~# lsusb

Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]

I'm using Kali 1.0.5 with the original kernel which is 3.6.11-cutdown:

root@kali:~# uname -r
3.6.11-cutdown

I've read that the since kernel 3.0 and higher a driver called rtl8192cu should be available that is supporting the chipset RTL8188CUS. After searching for "*8192*" I've found the module in kali:

root@kali:~# find / -name "*8192*" -print
/sys/bus/usb/drivers/rtl8192cu
/sys/module/8192cu
/sys/module/8192cu/drivers/usb:rtl8192cu
/opt/metasploit/apps/pro/ui/db/migrate/20130208192816_add_hidden_to_task_chain.rb
/lib/modules/3.6.11-cutdown/kernel/drivers/net/wireless/rtl8192cu
/lib/modules/3.6.11-cutdown/kernel/drivers/net/wireless/rtl8192cu/8192cu.ko
/lib/firmware/rtlwifi/rtl8192defw.bin
/lib/firmware/rtlwifi/rtl8192cfwU.bin
/lib/firmware/rtlwifi/rtl8192cfw.bin
/lib/firmware/rtlwifi/rtl8192cfwU_B.bin
/lib/firmware/rtlwifi/rtl8192cufw.bin
/lib/firmware/rtlwifi/rtl8192sefw.bin
/lib/firmware/RTL8192E
/usr/share/exploitdb/platforms/php/webapps/18192.txt

As no driver was loaded automatically, I've tried to load the module manually:

root@kali:~# modprobe 8192cu 

root@kali:~# lsmod

Module                  Size  Used by
ipv6                  207600  12
8192cu                411588  0
leds_gpio               1668  0
led_class               1788  1 leds_gpio

So the driver is loaded and now we can bring up the interface:

root@kali:~# ifconfig wlan0 up

root@kali:~# ifconfig wlan0
wlan0     Link encap:Ethernet  Hardware Adresse 80:1f:02:b3:50:8b
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

You can make a quick scan of available wifi networks via:

root@kali:~# iwlist wlan0 scan

Now you can either use the graphical WiFi manager or you can configure your WiFi via terminal, which is what I did. Just add the following lines to your /etc/network/interfaces:

auto wlan0
iface wlan0 inet dhcp
wpa-ssid <your SSID here>
wpa-psk <your wlan-key here>

After restarting the network you should have a working wifi connection.

root@kali:~# service networking restart

Note:
If you're using a hidden SSID you should read the following thread.

Additional, useful Unix tools in Kali via apt-get

Hey there,

just today I found a new useful linux command called "mtr", ok this tool is available since the late 90s of the last century, but for me it was new. It is an enhanced traceroute and is much quicker than traceroute, as it combines traceroute with ping and you can gather much more information with mtr than with traceroute. As I also install some more useful Unix commands via apt-get on my Kali Linux for Raspberry Pi, I just give a short overview about them (also as reminder for me):

- mtr (as explained above, much more powerful than traceroute)
- htop (nicer view than the normal top)
- dstat (nice view of resource  consumption with timestamp, e.g. dstat --tclmgry)
- tree (shorter and much powerful version of "find . -type d")
- links (if you need a browser in the shell; it is no fun to surf the web in a cli, but sometimes it can be useful)
- bc (little calculator in the shell)
- colordiff (you can guess it by the name, it enhances diff by adding color)
- tmux (alternative to screen)
- vim (no, I don't use emacs ;-)

A really great tool is tmux, that makes you're life in the shell much more easy. You should read the FAQ of OpenBSD to tmux for a quick'n'dirty introduction in it.

If you have any commands that are also useful for you regarding pentesting or to work more efficiently just leave a comment.

Another very useful tool for Mac OS X, regarding ssh is csshX. You can install it easily via homebrew on your Mac and you can manage different ssh sessions at once and you have also a master window that sends all input to every ssh session. Pretty neat.