Cupfighter.net

While doing a performance test on one of our customer environments we observed the impact of TCP offload and “Receive Side Scaling” (RSS) settings on the interface card on Windows web servers in combination with traffic handling.

Setup:

1. 2x Mercury Load Runner generators hitting public URL of customer

2. Served by 3x Windows2003 SP2 servers, running IIS6

3. Load being balanced by Cisco CSS11503 to web farm.

The CPU performance graph of the web servers with TCP offload and RSS enabled on the internet facing (FRONT) interface:

Similarly but a more outdated graph even more clearly showing that traffic is alternating from one web server to another:

Most interesting right!?

What makes this traffic to alternate if the load balancer has been set up to distribute the load evenly across the farm resp each Load Runner vuser to clear its cookies and session cache after each request?

We then stumbled over this read, knowing that TCP offload to network card is a classic one , but still:
http://blogs.msdn.com/psssql/archive/2010/02/21/tcp-offloading-again.aspx

And found out the characteristic that when TCP offload and RSS were disabled, the load is more evenly spread across the web farm:

I find this pretty cool.

Any comments?

As part of my work I was installing a Microsoft PKi infrastructure with two tiers. A root CA and an issuing CA.

Since the root CA is in another domain then the issuing CA, it took some fiddling and tweaking around with my CDP and AIA extensions, but that is another blogpost all together.

I knew I was in for some fun when when the following happened:

• I installed my Issuing CA and generated the certificate request
• I issued the request to my Root CA and generated the Issuing CA certificate
• I tried to install the Issuing CA certificate and got the following error:

Cannot verify certificate chain. Do you whish to ignore the error and continue? The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2168885613)

My first reaction was to call one of the network guest and notify him that I needed http access to the Issuing CA to the CDP location. But whil on the phone, I decided to try and to my surprise I was actually able to manually pull down the crl.

Intregued, I decided to check a few things:

• I could download the CRL from both CDP locations with Internet Exporer
• I could open the downloaded CRLs
• I could telnet to port 80 of the both webservers
• I could telnet to port 80 manually issue the GET /crl/CRLname.crl HTTP/1.0 command and get data back

O.K. what is going on here… Lets open PKI view, which is now included in Windows 2008 and Vista and can be downloaded for Windows 2000 and 2003.

It seemed that PKI view as in agreement, it too could not download the CRL from the CDP location

PKI view shows "Unable To Download" for both CDP locations

This did sent me on a wild goose chase:

• Microsoft own documentation, clearly blames it on unavailability of the CDP location, something I, by now, had triplle checked four times and refused to believe
• This “Network Builders” forum and many others, simply suggest to turn off revocation checking, but that is clearly not a worthy solution either.
• Apparently there is also an issue with serving delta CRLs threw IIS because the + sign at the end of the basename of a delta CRL file leads to so called “double escaping”. I could rule this out by looking at the IIS logs.
• In the end this technet forum post, about OCSP reponders Brian Komar points out:

But, as stated, I would use certutil to get the “best” answer on how is my configuration.
Certutil -verify -urlfetch “certfile.cer” will check *every* CDP and AIA URL (including OCSP) and tell you how they are all doing *at that specific instance in time” since it goes to the URLs immediately.
Brian

I exported the Issuing CA certificate from the certificate database of the Root CA and ran the command against is and this is what I found

E:\>certutil -verify -urlfetch <certfile>.cer
Issuer:
CN=Root CA
Subject:
CN=Issuing CA
Cert Serial Number: 115d5f6400020000000b
<snip>

—————-  Certificate AIA  —————-
Verified “Certificate (0)” Time: 0
[0.0] http://IIS1.domain1local/crl/Root-CA.crt

Verified “Certificate (0)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crt

—————-  Certificate CDP  —————-
Wrong Issuer “Base CRL (13)” Time: 0
[0.0] http://IIS1.domain1.local/crl/Root-CA.crl

Wrong Issuer “Base CRL (13)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crl

<snip>
E:\>

So while PKI view and the other error messages I was getting all pointed to the most common cause, it actually turned out that the CRl did get downloaded, but was not cryptographically relevant to what the system believes is the Root CA certificate.

Root cause

Inspection of the CRLs generated and the Root certificates installed showed what had caused the problem. In order to test the CDP extensions I had reissued the Root CA certificate, causing the Root CA to have three active certificates. Each with a different key.

This CA has three CA certificates

When validating the Issuing CA certificate, validation would end at the last certificate issued, however the CA still signs its CRLs with the key pair of the first certificate.

I guess for me there is nothing left but to reinstall the entire chain.

Of course there was a warning, as all speakers must have done the last couple of years, about the `ending` of IPV4. We are running out of ip addresses, we’ve heard that before.

Here you will find a nice link of where Geoff Huston is predicting the end of time:http://www.potaroo.net/tools/ipv4/index.html 

And in fact, we cannot ignore this. It will happen. And I want to be prepared, so that’s why I attended this session. I cannot longer sit back and hoping this would only happen when I’m retired. (and the Dutch government is not helping as well, as they have decided to extend pensioning from 65 to 67 years..)

Windows has already implemented the IPV6 stack from 2003 (and XP sp2) onwards and IPV6 from Vista onwards is the preferred protocol by default. Of course you can disable this, but in Win2k8 IPV4 is built on the IPV6 stack, so even when you disable IPV6, you’re always able to ping your local-home-address (::1).

Something I found during my research: Exchange 2003 on Windows 2008 needs IPV6, unless you disable it via a reghack (http://msmvps.com/blogs/ehlo/archive/2008/06/12/1634433.aspx).

You need to understand the principles (doh…) but networking is a piece of cake with IPV6

IPV4 is all about routing, IPV6 is all about shouting, was a statement of Mark Minasi.

Motivators to use IPV6:

• China is knocking at the internet-door.
• All European car-manufacturers have agreed to implement IPV6 in their cars as the standart protocol for car applications. (so beware, breaking will done via commands transported via IPV6..)

I don’t want to get in detail here, plenty of explanation on the web, but the modern OS-es all are capable of doing IPV6, and certainly I will dive deeper into this.  

You should too.

Today I spent most of my time analyzing the Slowloris and Nkiller2 denial of service (DoS) tools together with my colleague Gert Kremer.

Slowloris (name after the slow moving primates is a httpd DoS tool written by RSnake of ha.ckers. It works by tying up the httpd worker processes by slowly sending more and more headers of an httpd request.

Nkiller2 is a TCP/IP DoS attack tool which was published in issue 66 of Phrack magazine. It works by tying up httpd worker processes by requesting a file then stalling, mimicking the behavior of a client with full TCP/IP receive buffers.

Cisco CSS is a load balancer produced by Cisco.

In nearly all of the infrastructures built by my employer Schuberg Philis, the web servers are located behind a load balancer. In most cases a Cisco CSS. Because some of our customers were worried, I set out together with my colleague Gert Kremer to see if having a CSS load balancer in front of the web server provides any protection.

Slowloris

First we just had to try and find out what Slowloris did with an unprotected Apache server. The first video shows what happens when you run slowloris against a webserver. The window on the top left shows the number of apache processes, the top right window shows the scoreboard. This shows what the http processes are actually doing. The bottom window shows the slowloris output.

Slowloris vs Apache (No load balancer)

When slowloris is using 100 sockets, you can see 100 httpd workers in state “R”, meaning it is reading requests. The same is the case when running with 200 and 250 sockets. When running with 300 sockets the apache worker processes pool is exhausted and the web server can no longer service requests.

Slowloris vs Apache behind a Cisco CSS load balancer

Slowloris is running against the webserver with 3000 sockets (should be more then enough). As you can see on the top two windows the load balancer does not forward any of the incomplete requests to the webserver. We have stress tested the loadbancer up to 10,000 sockets and it had no effect on the loadbancer.

NKiller

Nkiller vs Apache (No load balancer)

In the video we see for windows. Top left and right show the number of apache processes and the apache dashboard. The middle window displays the NKiller output and the bottom window TCPdump.

When NKiller starts we see the it exhausts the httpd workers processes by putting them in a state where they are hanging while writing their reply back to the client.
Nkiller vs Apache behind a CSS load balancer

When NKiller was used against a server protected by a Cisco CSS load balancer the packets received from the load balancer do not match the expections of the Nkiller tool and the tool crashed producing a segmentation fault.