When implementing Cisco Wireless network infrastructure Enno and Daniel got the impression that, security wise, these systems smell.

First part of the presentation focuses on what a typical implementation looks like.

There are three generations:
1.    Structured Wireless-Aware Networks (SWAN)
2.    Based on managed APs and LWAPP (After acquiring Airport)
3.    Cisco Unified Wireless Network

The talk focuses on generation one and three.

The are a couple of attack paths: traffic in transit, cryptographics and against components.

First up is SWAN. It mainly runs on WLCCP protocol messages, this protocol is proprietary, so the patents are needed to discover the inner workings and the deviations from the patent.

The key management is arranged by Cisco’s proprietary key management framework called Cisco Centralized Key Management (CCKM). This framework allows the key material for clients from one access point to the other.

One of the properties of the protocol is the selection of the WDS Masters that controls all communication between the APs.
He communication  between the APs is authenticated by means of LEAP. The security of LEAP is debatable at best. And Cisco’s fix, deriving two additional keys based on the first key is debatable too.

Management interfaces are the Achilles’ heel of many systems.

So what do you need for a practical attack against APs? If you can get to the AP’s management interface, you can identify it by identifying WLCCP speakers, sniff the intra AP traffic and crack the LEAP secret. Then you can evict the WDS master if necessary.

Daniel next demoed the attack. He used Loki to sniff the backbone interface to identify the WDS master. Loki can now be used to create a new WDS master but inserting a new WDS master. The master priority is configurable up to 254, but the protocol can handle a value to 255, so you can always win this election.
Next Loki can be used to brute force the detected WDS password and the revealed password can be used to derive the additional security keys.

Even though there are some parts of the crypto space that smells, Enno and Daniel where not able to find practical exploits here.

Management interfaces however are another story.

SNMP is a good friend, especially if people forget to reset their community strings. The SNMP interface does not allow you to reset passwords of existing users, but it does allow you to create administrative users.

The web interface of Cisco WLAN management tooling is web based, with all the classical web based attacks like Cross Site Scripting.

Enno demoed a web based attack. Intercepting a request to the web based interface with burpsuite and rewriting the request he was able to trigger a buffer overflow in the wireless management appliance. This makes you wander what would happen if you run a fuzzer against it.

Key points to take away:
•    “Enterprise WLAN solutions” might be complex beasts
•    There many be not so obvious vulnerabilities
•    Use common sense when deploying
•    The problems outlined are not Cisco specific

The majority of problems are based on management interface. They should never be publicly exposed.

Since the root CA is in another domain then the issuing CA, it took some fiddling and tweaking around with my CDP and AIA extensions, but that is another blogpost all together.

I knew I was in for some fun when when the following happened:

• I installed my Issuing CA and generated the certificate request
• I issued the request to my Root CA and generated the Issuing CA certificate
• I tried to install the Issuing CA certificate and got the following error:

Cannot verify certificate chain. Do you whish to ignore the error and continue? The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2168885613)

My first reaction was to call one of the network guest and notify him that I needed http access to the Issuing CA to the CDP location. But whil on the phone, I decided to try and to my surprise I was actually able to manually pull down the crl.

Intregued, I decided to check a few things:

• I could download the CRL from both CDP locations with Internet Exporer
• I could open the downloaded CRLs
• I could telnet to port 80 of the both webservers
• I could telnet to port 80 manually issue the GET /crl/CRLname.crl HTTP/1.0 command and get data back

O.K. what is going on here… Lets open PKI view, which is now included in Windows 2008 and Vista and can be downloaded for Windows 2000 and 2003.

It seemed that PKI view as in agreement, it too could not download the CRL from the CDP location

PKI view shows "Unable To Download" for both CDP locations

This did sent me on a wild goose chase:

• Microsoft own documentation, clearly blames it on unavailability of the CDP location, something I, by now, had triple checked four times and refused to believe
• This “Network Builders” forum and many others, simply suggest to turn off revocation checking, but that is clearly not a worthy solution either.
• Apparently there is also an issue with serving delta CRLs threw IIS because the + sign at the end of the basename of a delta CRL file leads to so called “double escaping”. I could rule this out by looking at the IIS logs.
• In the end this technet forum post, about OCSP reponders Brian Komar points out:

But, as stated, I would use certutil to get the “best” answer on how is my configuration.
Certutil -verify -urlfetch “certfile.cer” will check *every* CDP and AIA URL (including OCSP) and tell you how they are all doing *at that specific instance in time” since it goes to the URLs immediately.
Brian

I exported the Issuing CA certificate from the certificate database of the Root CA and ran the command against is and this is what I found

E:\>certutil -verify -urlfetch <certfile>.cer
Issuer:
CN=Root CA
Subject:
CN=Issuing CA
Cert Serial Number: 115d5f6400020000000b
<snip>

—————-  Certificate AIA  —————-
Verified “Certificate (0)” Time: 0
[0.0] http://IIS1.domain1local/crl/Root-CA.crt

Verified “Certificate (0)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crt

—————-  Certificate CDP  —————-
Wrong Issuer “Base CRL (13)” Time: 0
[0.0] http://IIS1.domain1.local/crl/Root-CA.crl

Wrong Issuer “Base CRL (13)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crl

<snip>
E:\>

So while PKI view and the other error messages I was getting all pointed to the most common cause, it actually turned out that the CRl did get downloaded, but was not cryptographically relevant to what the system believes is the Root CA certificate.

Root cause

Inspection of the CRLs generated and the Root certificates installed showed what had caused the problem. In order to test the CDP extensions I had reissued the Root CA certificate, causing the Root CA to have three active certificates. Each with a different key.

This CA has three CA certificates

When validating the Issuing CA certificate, validation would end at the last certificate issued, however the CA still signs its CRLs with the key pair of the first certificate.

I guess for me there is nothing left but to reinstall the entire chain.

Yesterday we discovered an issue with Windows 2008 clusters: manually added persistent routes disappear from the active routes table, when taking offline (or failing over) a cluster group containing an ip-address-resource.

This issue is documented here. This same article also describes a workaround for when you have multiple gateways on multiple NIS’c.

By changing your route add command from e.g. <route add 10.1.0.0 mask 255.255.255.0 10.1.0.1 –p> to <route add 10.1.0.0 mask 255.255.255.0 0.0.0.0 if 25>

With this second command you bind the route to the interface instead of an ip-address. And since it is now bound to a local device any cluster failover will leave the route in the routing table.

However this will not solve the issue we discovered yesterday: We are using 2 gateways ‘behind’ the same interface. So binding the route to the interface will not help here.

Example interface 18: 192.168.251.36 mask 255.255.255.0 192.168.251.1, with added route 192.168.250.0 mask 255.255.255.0 192.168.251.3 –p.

When an ip-address will be taken offline (fails over) the Active route 192.168.250.0 255.255.255.0 192.168.251.3 will be removed.

Accidentally we found out that adding the interface to the route will solve this new issue (thanks our collegue Enrico). So our new route command will have to look like this:

<Route add 192.168.250.0 mask 255.255.255.0 192.168.251.3 if 18>. This will leave the route in the active routes table.

Why does this work? And is it reliable?

Since we couldn’t find any google/Microsoft hits on this particular issue, we had to do a little registry digging.

The standard command <Route add 192.168.250.0 mask 255.255.255.0 192.168.251.3 > just adds the persistent route to the registry which triggers the ‘bug’.

However the new command <Route add 192.168.250.0 mask 255.255.255.0 192.168.251.3 if 18> also makes 14 changes in the cluster part of the registry telling it that this route is bound to the adapter and to be left behind on the local server in case of a failover

So I think it look pretty reliable. We did lots of reboots and failovers on the cluster and the routes seem pretty persistent now..

Internet was provided by XS4ALL, BIT and OpenTransit. There were direct peering connections with Akamai, Google and Giganews.

First problem: how do you get from Vierhouten to Amsterdam? In Vierhouten you have several options:
•    3KM fibers to Nunspeet
•    There are two fibers of KPN and UPC in Vierhouten

KPN provided a link from Amsterdam to Putten, Putten to Harderwijk, Harderwijk to Hunnen, Hunnen to Uddel, Uddel to Vierhouten. Last 650m was put there by the NOC team themselves.

Cables used:
•    560m SMF
•    3.5 KM MMF
•    15KM Cat 5e
•    5Km rope.

One of the biggest challenges: putting fiber in trees.

Network equipment needed:
•    1 * Foundry MLX 8 – Core
•    4 * Mach 4002 – Distribution
•    44 * HP Procurve – Access

Dixie toilets acted as Datenklos because they can be locked and are watertight. A datenklos lock can be picked in less than 5 seconds.

The NOC team did not receive a single abuse call during the entire event.

Nice hacks during HAR2009:
•    WPAD. Set via DynamicDNS to wpad.visitors.har2009.net and you will get a lot of traffic for his proxy server.
•    One report of a MitM attack in the lounge

Wireless equipment:
•    30 Cisco 1131 ap’s
•    30 Cisco 1242 ap’s
•    2 WLC 4404 controllers
•    1 WCS Server
•    1 Location appliance
Total value 170,000 Euro’s

The Location appliance can be used to locate users. There were about 1500 unique users and a peak of 700 simultaneous users. There were quite a few rogue wireless access points.

Slowloris (name after the slow moving primates is a httpd DoS tool written by RSnake of ha.ckers. It works by tying up the httpd worker processes by slowly sending more and more headers of an httpd request.

Nkiller2 is a TCP/IP DoS attack tool which was published in issue 66 of Phrack magazine. It works by tying up httpd worker processes by requesting a file then stalling, mimicking the behavior of a client with full TCP/IP receive buffers.

Cisco CSS is a load balancer produced by Cisco.

In nearly all of the infrastructures built by my employer Schuberg Philis, the web servers are located behind a load balancer. In most cases a Cisco CSS. Because some of our customers were worried, I set out together with my colleague Gert Kremer to see if having a CSS load balancer in front of the web server provides any protection.

Slowloris

First we just had to try and find out what Slowloris did with an unprotected Apache server. The first video shows what happens when you run slowloris against a webserver. The window on the top left shows the number of apache processes, the top right window shows the scoreboard. This shows what the http processes are actually doing. The bottom window shows the slowloris output.

Slowloris vs Apache (No load balancer)

Click here to view the embedded video.

When slowloris is using 100 sockets, you can see 100 httpd workers in state “R”, meaning it is reading requests. The same is the case when running with 200 and 250 sockets. When running with 300 sockets the apache worker processes pool is exhausted and the web server can no longer service requests.

Slowloris vs Apache behind a Cisco CSS load balancer

Click here to view the embedded video.

Slowloris is running against the webserver with 3000 sockets (should be more then enough). As you can see on the top two windows the load balancer does not forward any of the incomplete requests to the webserver. We have stress tested the loadbancer up to 10,000 sockets and it had no effect on the loadbancer.

NKiller

Nkiller vs Apache (No load balancer)

Click here to view the embedded video.

In the video we see for windows. Top left and right show the number of apache processes and the apache dashboard. The middle window displays the NKiller output and the bottom window TCPdump.

When NKiller starts we see the it exhausts the httpd workers processes by putting them in a state where they are hanging while writing their reply back to the client.
Nkiller vs Apache behind a CSS load balancer

Click here to view the embedded video.

When NKiller was used against a server protected by a Cisco CSS load balancer the packets received from the load balancer do not match the expections of the Nkiller tool and the tool crashed producing a segmentation fault.