Check Point Security Gateway R75.20 HFA 20 … *SLASH* … A Tale of Checkpoints mysterious disappearing packets … ‘Negated’

Morning … so we had a new layer 7 application rolled out on our network. This application pretty much used http and https POST, GET, packets to communicate and it wasn’t working. This always creates a lovely little finger pointing vacuum which sucks everyone in. Needles to say our engineers were saying ‘we’ are not blocking it’ meaning our firewalls were not blocking any packets transmitted from the sources specified on the ports defined. OK.

Back to the application vendor, round and round we went for hours, testing, tracing, and nadda, we have no evidence to say its being blocked. OK.

It got escalated, standard.

So I wanted to see for my self what was going on, placed a few calls T’d it up with the vendor, and away we went.

My laptop connected locally to domain and network. Installed application. SSH’d to firewall and did tcpdump on interface my traffic would come in on. Booted up wireshark and set to log local interface. Application did its ‘thing’ now this is where I became stumped. Packets flew out my laptop, and never came in firewall interface (or so I thought). After some discussion, I put a call in to a friend as I was under the impression the tcpdump command dumped ALL traffic it saw … apparently not.

Basically the IPS was dropping the non_compliant_HTTP requests despite an IPS exclusion which one of our engineers had setup correctly. This wasn’t displaying in the smart view tracker or a TCP dump I did on the internal interface of the firewall via SSH. The only way I manage to see the packets drop was by doing a top level debug on the firewall via ssh grep’d to my source IP. Which displayed the below

fw-lambeth-1[admin]#
fw-lambeth-1[admin]#
fw-lambeth-1[admin]# fw ctl zdebug + drop | grep 10.10.10.10
Dec  2 10:01:19 fw-lambeth-1 <kern.[LOG_CRIT]> kernel: FW-1: Initializing debugging buffer to size 1023K
;fw_log_drop: Packet proto=6 10.10.10.10:54861 -> 8.8.8.8:80 dropped by fwpslglue_chain Reason: PSL Reject: HTTP_PSL;
;fw_log_drop: Packet proto=6 10.10.10.10:54861 -> 8.8.8.8:80 dropped by fwpslglue_chain Reason: PSL Reject: HTTP_PSL;
^Cfw-lambeth-1[admin]#

NB. IP addresses changed for obvious reasons.

I validated this in a wire shark packet capture and saw the packets so I knew my laptop was sending them at this point I basically just told support provider to go straight to check point and get me the hotfix (I found the one we needed on a KB article) then they sent me the one for Secure OS which I luckily checked before hand through CPVINFO command and hat to wait an hour for Checkpoint in Israel to send the correct one for IPSO (checked it again!) applied.

This whole process took around 24 hours. You can find release notes below.

Release Notes for Hot Fix fw1_wrapper_HOTFIX_FOXX_HF_HA20_480 build 983480002_2 Linux

=====================================================================================
Overview
========

Hotfix should be installed on Gateway
This Hot Fix should be installed over Check Point Security Gateway R75.20 HFA 20
Files updated by this hotfix (Linux):
=====================================
please verify that in the cpvinfo output on the below files you see :
Module Name = fw1
Minor Release = foxx_hf_ha20_480
Build Number = 983480002
/opt/CPsuite-R75.20/fw1/boot/modules/fwmod.2.6.18.cp.i686.o
/opt/CPsuite-R75.20/fw1/boot/modules/fwmod.2.6.18.cp.i686.noPAE.o
/opt/CPsuite-R75.20/fw1/boot/modules/fwmod.2.6.18.i686.o
/opt/CPsuite-R75.20/fw1/boot/modules/fw6mod.2.6.18.cp.i686.o
/opt/CPsuite-R75.20/fw1/boot/modules/fw6mod.2.6.18.cp.i686.noPAE.o
/opt/CPsuite-R75.20/fw1/boot/modules/fw6mod.2.6.18.i686.o
/opt/CPsuite-R75.20/fw1/lib/inspectEngine.2.6.18.cp.i686.o
/opt/CPsuite-R75.20/fw1/lib/inspectEngine.2.6.18.cp.i686.noPAE.o
/opt/CPsuite-R75.20/fw1/lib/inspectEngine.2.6.18.i686.o

To verify that the fix was installed correctly,  run ‘cpvinfo’ utility on each of the modified file.

For Example:
# cpvinfo <file_name>
And make sure that the Module Name, Minor Release and Build Number  have been changed according to the output above

Bug Fixes:
==========
CR01148828 – Quick UFP cause Block HTTP Non Compliant error when access legitimate websites (google, checkpoint, microsoft…)
CR01149768 – BUG: soft lockup – CPU#0 stuck for 10s!
Hotfix Installation Instructions:

=================================

Please not this firewall is now decommissioned which has allowed me to post this article.
Hope this helps someone
Marc

Managing Multiple domains

So I needed to manage multiple domains from a machine which is not on any of them. How can I run multiple MMC consoles without being on any of them? BTW this is in a Windows Active Directory environment only.

Lets say the domains are as follows

London.co.uk
Dev.London.co.uk
Manchester.co.uk
Birmingham.co.uk
Dev.Birmingham.co.uk
NB. All domains are accessible from an layer 3 (IP) perspective either via VPN’s, direct links, or static routing.

Firstly you have to tell the machine where one of the Domain Controllers (DC) are so it knows where to send requests? This is done using the host file on windows. This article is not going to go into detail about this, but see below for entries into host file.

192.168.10.10    London.co.uk
192.168.20.20    Dev.London.co.uk
192.168.30.30    Manchester.co.uk
10.10.40.40        Birmingham.co.uk
10.10.50.50        Dev.Birmingham.co.uk

Once DNS has been refreshed or the system restarted you have the ability to run this command.

/NETONLY – http://technet.microsoft.com/en-us/library/bb490994.aspx

This command allows you to run as a user form that domain when not on the domain.

runas /netonly /user:domain\username \”mmc.exe dsa.msc /server=192.168.10.10″ <– if wanting to connect to London.co.uk

This can be added to a batch file to request the username as shown below

@echo off
set /p id=”Enter Username: ” %=% runas
/netonly /user:london.co.uk\%id% “mmc.exe dsa.msc /server=192.168.10.10”

It does however mean having a batch file per domain but once configured allows you to switch between different environments without having to jump onto servers in that environment. Anyway this made my life easier and hope it helps someone else.

Marc

PAC files (… yes Proxy Auto Configuration Files)

What is a PAC file?
A PAC file is a Proxy Auto Configuration file, this defines how web browsers access the internet and can automatically set parameters based on URL, Service, Source and Destinations. The language is Java Script and it is designed in such a way which it can cater for any scenario.

How to implement?
Manual – Manually setting the file location in browser (see local testing below)
Enterprise wide – Set PAC file using something like group policy or similar
WDAP – Web Proxy Auto Discovery protocol (can also be set by Group Policy)

There are advantages and disadvantages to each of the above and depends on what suits your enterprise better.

Local Testing
Windows
You can set the PAC file locally and test this
1. Copy the PROXY.PAC file to the C:\WINDOWS directory, or other directory of your choice.
2. In the browser proxy settings, configure the Use Automatic Configuration Script (IE) URL to: Internet Explorer, use: file://c:/windows/proxy.pac

Linux
$ sudo apt-get install libpacparser1
$ cd /tmp/
$ wget http://pactester.googlecode.com/files/pactester-1.0.8.tar.gz
$ tar xvf pactester-1.0.8.tar.gz
$ mkdir $HOME/pactester
$ cd pactester-1.0.8/build/
$ ./install.sh $HOME/pactester

Usage:
pactester -p /path/to/proxy.pac.file -u url
pactester -p /path/to/proxy.pac.file -u url -c client.ip.address.here

Sample 1
Showing how to set internal network ranges for general bypass rules. The ability to set different proxy servers for different service types. Using multiple proxy servers for resilience.

function FindProxyForURL(url, host)
 { if (isInNet(host, "192.0.0.0", "255.0.0.0")) { ## defines internal LAN
 return "DIRECT"; ## states to go direct if not a part of 192.0.0.0
 }
 else
 { if (shExpMatch(url, "http:*")) ## tells to use proxy for http protocol
 return "PROXY 192.168.0.1:8080;
 PROXY newvirtualproxy:8080; PROXY 192.168.0.1:8080" ;
 if (shExpMatch(url, "https:*")) ## tells to use proxy for https protocol
 return "PROXY 192.168.0.1:8080;
 PROXY newvirtualproxy:8080; PROXY 192.168.0.1:8080" ;
 if (shExpMatch(url, "ftp:*")) ## tells to use proxy for ftp protocol
 return "PROXY 192.168.0.1:8080;
 PROXY newvirtualproxy:8080; PROXY 192.168.0.1:8080" ;
 return "DIRECT"; ## tells to go direct if not http, https or ftp
 } }

Sample 2

Showing how to set exclusions to the overal proxy rules based on specific URL’s. Setting network exclusion ranges.

function FindProxyForURL(url, host)
 { if (isPlainHostName(host)) { return "DIRECT"; }
 if (shExpMatch(host, "127.*")) { return "DIRECT"; } ## tells to go direct for sites that start with 127.
 if (shExpMatch(host, "192.*")) { return "DIRECT"; } ## tells to go direct for sites that start with 192.
 if (shExpMatch(host, "testdomain.com")) { return "DIRECT"; } ## example of url bypass
 if (shExpMatch(host, "my.sub.domain.testdomain.com")) { return "DIRECT"; } ## example of url bypass
 if (shExpMatch(host, "*.secure.*")) { return "DIRECT"; } ## example of url bypass
 if (shExpMatch(host, "portal.testdomain.com")) { return "DIRECT"; } ## example of url bypass
 if (isInNet(myIpAddress(), "192.168.2.0", "255.255.255.0")) { return "PROXY 192.168.1.253:8080"; }
 ## Setting specific proxy for network ranges
 if (isInNet(myIpAddress(), "10.0.2.0", "255.255.255.0")) { return "PROXY 10.0.2.253:8080"; }
 ## Setting specific proxy for network ranges
 else
 { if (shExpMatch(host, "*.jp")) { return "DIRECT"; } ## setting bypass for TLD .JP
 if (shExpMatch(host, "*.gmail.com")) { return "DIRECT"; } ## Setting bypass for everything Gmail
 return "PROXY proxy01.com:8080";

Sample 3

Showing how to simply the above using a different method

function FindProxyForURL(url, host)
 { if (isInNet(host, "192.0.0.0", "255.0.0.0")) { return "DIRECT"; } ## Networks to bypass & go direct
 else
 { if (shExpMatch(url, "http:*")) return "PROXY proxy01.com:8080" ; ## setting http proxy
 if (shExpMatch(url, "https:*")) return "PROXY proxy02.com:8080" ; ## Setting https proxy
 if (shExpMatch(url, "ftp:*")) return "PROXY proxy03.com:8080" ; return "DIRECT";} } ## Setting ftp proxy

Summary

These are meant to help others design and script their own, there are many examples above on different ways to do the same thing, to allow you to have more granular flexibility for bypass rules which are unavoidable sometimes. If you think about this now and plan to have a process to have exclusions even if this is based on groups in LDAP, this will save you time and effort in the future and add a level of resilience to the oh so sacred Internet Pipe!

Links
Issues with IE 5.x
Online Java debugger
Command Reference

Java Vs Andriod

Java originally came to the market courtesy of James Gosling who worked at Sun Microsystems in 1995. Since then it has become synonymous with most electronic components which need to run software when the vendors do not have there own. This has become more popular due to the costs involved in maintaining your own operating system. Java was and still is very powerful software which can run on a number of different devices regardless of computer architecture, it was designed so that programs can be loaded over a network and run locally making it very versatile and extreme cross platform allowing everyone to be able to lean on Java.

This makes it a viable choice for manufacturers who need software to operates their devices but do not have any of there own. The reason why this works so well is because Java is regarded as a low-level language which means it can interact with the hardware closely and means you can keep it lightweight and portable. Java works on devices from high end computers, so low end mobile phones and digital cameras allowing them all to function and work using the same version of Java. Devices initially had limited GUI’s which didn’t impact Java much as it didn’t have one.

In Steps Android, ‘the new kid on the block’ …

Android is a Linux based operating system which was created by Android Inc, financed by Google and then purchased by them in 2005. The first official version of Android was released in 2007 and the first phone came pre-loaded with it in 2008. This was a pinnacle moment for entering the mobile market before smart phones really took off to allow Andriod to test, tweak, and flush out as many bugs as they could (they didn’t flush out as many as they wanted!). It seems like there s drive to make Android a generic operating system which can be used across many different hardware components, regardless of manufacturer. This is a similar operating model to Java but the main difference being one has a GUI and one doesn’t which when entering a market of having everything connected, all the time with higher resolution screens and brighter colours means a GUI’s is make or break.

I see new devices coming out every day, cameras, fridges, phones, tablets, smart watches, car entertainment systems, media systems, and even a head mounted display for ski googles (Google Glasses concept idea?). Soon it won’t be a case of, is your phone running Android or not, the questions will be is your kitchen running Android or not. If Google can sort out the security of Android (as Jail breaking becomes common place) so it can be used securely then this could be the new Java. Target Android for apps, and services and when the ‘white goods’ market hits and every electronic device is ‘connected’ this is when the demand for apps will sky rocket. I hope the industry will put some standards around this for developers and manufacturers. If they do we will have the chance to see some amazing apps, and technology to come. If not you will just be running away from your toaster after some 13 year old kid roots it remotely from his bedroom in his pajamas while drinking a cup of tea. Only time will tell … Sunday morning brain dump …

The views expressed here are my own and do not represent any influence on my professional work.
Regards
Marc

Virgin Media ‘Super’ Hub … Myths

So i decided to refresh my internet connection (it had been in place for over 3 years). I also had a notification from Ofcom saying the Wandsworth Exchange had been upgraded to Fibre. So lets see who can offer Fibre from there … turns out to be just BT (Infinity) and Virgin (Virgin Media). So deciding which is the lesser of two evils i went with Virgin.

Yes i had been told about the heinous traffic shaping, and throttling they do but I went ahead with this anyway. I had also recently ordered Virgin Big Red Business for work purposes so i was aware of how it worked and the problems around the ‘Super’ Hub (although not how many problems or bad they were until now!).

So current setup

BT Phone Line – £14.00

BeThere Broadband – £24.95

Total – £38.95

Download – 7 mb / Upload – 1.2 mb (wired)

So initial virgin only took 6 working days to install due to existing cabling which was already in place. So after 6 days i got my kit, connected it all up and away i went. After configuring all the devices to connect onto the LAN. So i begin with a simple speed test thanks to http://www.speedtest.net

Download 33 mb / Upload 7 mb (wired)

Download 12 mb / Upload 5 mb (wireless)

Wasn’t the type of performance i was expecting but i thought lets give it a bit of time and review in a week. A week passed and the same results returned. Off to google and the forums i went … so hours later i find out a number of things

I am running the latest Firmware of the re-branded Netgear R36 … this had removed alot of the functionality i was alooking for. Port forwarding, DynDNS, SSH direct to device.

In summary Known problems

-Performance of box over cabling connection

-Extremely poor wireless connection

-Locked down functions preventing users from basic tasks

Anyway i decided to go down the official road for this instead of the normal research, hack/fix/brake one. They explain they are going to send me a new device, and i agree but state an extension to my cancellation period and if this doesn’t work they say they are going to send me a D-Link and request i put the device in to Modem Mode. (ill come onto this shortly). The lady also explained that there are quite a few errors on the circuit.

SO while i wait for this, i begin to look into the modem mode function of the device, so i re-connect my old Linksys WAG54G which everyone has one of these lying around if you are an IT geek, don’t ask me why either! Anyway after of troubleshooting this fixes the wireless problem but the actual speed is still poor.

So the very next day i receive the new hub (i missed the first attempt at delivery as they gave me a text 15 minutes before it arrived midday on a Wednesday when i was at work. I have a quick flick through the doco and it has a number of warning about a new power supply, i check this against the old one and yes the voltage is still the same but the amps has increased. I connect it all up and register the mac with Virgin Media, i did notice the old firmware, and so i tested ssh and dyndns beforehand which all worked fine, and the menu option are very similar to the NetGear generation routers which have more functions (shame this has been removed).

The MAC addresses between the old and new devices are very different suggesting there has been a new generation of device which doesn’t have the previous problems. If your device has a MAC beginning with the following

MAC: 4c60DE****** (Netgear)

Then i would call Virgin and request a new ‘Super’ Hub. If this begins with a

MAC: 00********** (Netgear Inc)

Then you already have one. From a MAC lookup online it says different company names again suggesting different generations of the device.

Now i jump back onto http://www.speedtest.net and below are the results

Download 101 mb / Upload 10 mb (wired)

Download 26 mb / Upload 11 mb (wireless)

Ok this is more like i was expecting. But the wireless is still poor, and they automatically upgrade the device to the latest firmware. SO the long and short of it as, if you got Virgin ‘Super’ Hub with a MAC of 00********** your actual connection should be fine and your speed reflect this. But the wireless is still poor (this might change in future revisions of the firmware) but you would probably be better off doing it with a device of your own if you can be bothered to set it up. One other little thing, it only has one Antenna in it so you can either broadcast on 2.4 GHz (802.11a/b/g) or 5 GHz (802.11n) meaning if you have devices which don’t support 802.11n then leave it on 2.4 GHz.

If anyone knows a way of using DynDNS directly off the ‘Super’ hub drop me a line i’d love to know how!

I hope this helps someone as I bet there are loads of people out there on the old devices paying for a service they are only getting half of!

References

Test your ISP for traffic shaping

http://broadband.mpi-sws.org/transparency/bttest.php

Test your internet speed from different locations

http://www.speedtest.net/

MAC address company lookup

http://aruljohn.com/mac.pl

Sky Go on iPad

So Sky have patched there iPad again to not allow jail broken devices. Cydia have a patch.

Http://n00neimp0rtant.dyndns.com/repo

Dig DNS tools comes to iPad

So I was browsing the app store and turns out DIG is available for iPad … Now that will come in handy. Although I have been finding different DNS servers behave differently. I’ve had this before on desktops machines and so now have preferred servers to check with. This is only related to doing any queries all cname, mx, a, etc all work fine but sometimes you want the works.

Marc