Click here to view the embedded video.

We are always looking for more Cupfighters.

My main objective in watching the webcast (which is not my usual habit) was to find out if systems that have the described workaround applied still need to apply the patch. The webcast did not give a definitive answer but this YouTube video and the Netifera website and the twitter accounts Thai Duong provide the answer: Yes you should apply the patch a.s.a.p!

Click here to view the embedded video.

However the Q&A section of the talk did give me, as a security operations guy, quite some food for thought. I made some notes in my own Twitter feed, which I have summarized here.

Q: Why did Microsoft release and OOB update for a vulnerability rated “only” as important?
A: The vulnerability itself is rated as Important because it is not a vulnerability that directly leads to remote code execution on the vulnerable system, however exploitation of the vulnerability will lead to disclosure of all information in the webroot including web.config. This information can be used for session hijacking, compromising backend databases and to attack associations between websites, e.g. the association of a website with PayPal. Hence an out of band patch was warranted.

Q: Why only release to the download center and not to WSUS etc?
A: We felt we needed to get this update out quickly, the people that need to apply this patch quickly are mainly enterprises who are capable of applying patches without the aid of WSUS. Developing the WSUS capabilities would add another few days of delay to the deployment of this patch.

Q: Is the attack actively used?

A: We have seen limited attacks against this vulnerability as well as continuous efforts to to bypass installed workarounds.

Q: Can the patch be uninstalled, does it require a reboot?
A: The patch can be uninstalled and does require a reboot.

Q: If you have multiple versions of .Net installed on the system, do you need to install all patches for each version of .Net?
A: Yes.

Q: If you have 64bit and 32bit version of Asp.Net installed, do you need to apply both 64bit and 32bit patches?
A: No, the 64bit patch will patch the 32bit versions as well.

Q: Should we regard the ASP.NET MachineKey as compromised?
A: Yes, if you have set a static MachineKey it is recommended to replace this key with a new key. (Information on AutoGenerated MachineKeys was not provided)

Q: Will the patch have an effect on end-users?
A: Yes, information stored on the client that is protected by the MachineKey can no longer be validated. This can e.g. mean that users whoo used a ‘remember me’ function will have to login in again.

Q: Does the patch need to be applied to all nodes of a cluster?
A: Yes, because the patch changes the way data in transit (such as e.g. viewstate) is encrypted, this patch needs to be applied to all nodes in a cluster as the same time or users may experience unexpected results.

Q: Does the patch change IIS?
A: No, the patch only changes ASP.NET, not IIS.

Q: Does the patch change the way encrypted data is stored on the server?
A: No, the patch changes the way data in transit is cryptographically protected, both encryption and signing is now applied. It does not effect any encrypted data stored on the server.

Q: Are the patches in the download center “smart” enough to know if they are applicable for the machine you apply them to?
A: No, detection capabilities will be built into the patches once they are deployed to WSUS.

Q: Should the update be applied to all .net installation, not just web servers?
A: The vulnerability only manifests itself via web servers. For now it is recommended to only install patches there, and way for the patches to appear in WSUS before patching other .net installs. But remember a system with an unpatched .net installation will become vulnerable as soon as a webserver is installed.

Q: Should the workaround be removed prior to patching?
A: No, you can apply the patch with the workaround in place. If you need to do so you can then remove the workaround after the patch has been applied. CustomErrors generally does not hurt and neither does UrlScan all though UrlScan is known to break SharePoint and may break other web applicaitons as well

Q: Do customer applications need to be recompiled?
A: No.

Scott Guthrie’s blog has an excellent overview of which patch is applicable to which platform.

Since the root CA is in another domain then the issuing CA, it took some fiddling and tweaking around with my CDP and AIA extensions, but that is another blogpost all together.

I knew I was in for some fun when when the following happened:

• I installed my Issuing CA and generated the certificate request
• I issued the request to my Root CA and generated the Issuing CA certificate
• I tried to install the Issuing CA certificate and got the following error:

Cannot verify certificate chain. Do you whish to ignore the error and continue? The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2168885613)

My first reaction was to call one of the network guest and notify him that I needed http access to the Issuing CA to the CDP location. But whil on the phone, I decided to try and to my surprise I was actually able to manually pull down the crl.

Intregued, I decided to check a few things:

• I could download the CRL from both CDP locations with Internet Exporer
• I could open the downloaded CRLs
• I could telnet to port 80 of the both webservers
• I could telnet to port 80 manually issue the GET /crl/CRLname.crl HTTP/1.0 command and get data back

O.K. what is going on here… Lets open PKI view, which is now included in Windows 2008 and Vista and can be downloaded for Windows 2000 and 2003.

It seemed that PKI view as in agreement, it too could not download the CRL from the CDP location

PKI view shows "Unable To Download" for both CDP locations

This did sent me on a wild goose chase:

• Microsoft own documentation, clearly blames it on unavailability of the CDP location, something I, by now, had triple checked four times and refused to believe
• This “Network Builders” forum and many others, simply suggest to turn off revocation checking, but that is clearly not a worthy solution either.
• Apparently there is also an issue with serving delta CRLs threw IIS because the + sign at the end of the basename of a delta CRL file leads to so called “double escaping”. I could rule this out by looking at the IIS logs.
• In the end this technet forum post, about OCSP reponders Brian Komar points out:

But, as stated, I would use certutil to get the “best” answer on how is my configuration.
Certutil -verify -urlfetch “certfile.cer” will check *every* CDP and AIA URL (including OCSP) and tell you how they are all doing *at that specific instance in time” since it goes to the URLs immediately.
Brian

I exported the Issuing CA certificate from the certificate database of the Root CA and ran the command against is and this is what I found

E:\>certutil -verify -urlfetch <certfile>.cer
Issuer:
CN=Root CA
Subject:
CN=Issuing CA
Cert Serial Number: 115d5f6400020000000b
<snip>

—————-  Certificate AIA  —————-
Verified “Certificate (0)” Time: 0
[0.0] http://IIS1.domain1local/crl/Root-CA.crt

Verified “Certificate (0)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crt

—————-  Certificate CDP  —————-
Wrong Issuer “Base CRL (13)” Time: 0
[0.0] http://IIS1.domain1.local/crl/Root-CA.crl

Wrong Issuer “Base CRL (13)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crl

<snip>
E:\>

So while PKI view and the other error messages I was getting all pointed to the most common cause, it actually turned out that the CRl did get downloaded, but was not cryptographically relevant to what the system believes is the Root CA certificate.

Root cause

Inspection of the CRLs generated and the Root certificates installed showed what had caused the problem. In order to test the CDP extensions I had reissued the Root CA certificate, causing the Root CA to have three active certificates. Each with a different key.

This CA has three CA certificates

When validating the Issuing CA certificate, validation would end at the last certificate issued, however the CA still signs its CRLs with the key pair of the first certificate.

I guess for me there is nothing left but to reinstall the entire chain.

On the 19th of November Frank Breedijk announced that Jason Mansfield, who runs the website http:/clinicallyawasome.com, has won the contest by sending in the name Seccubus. A bottle of Vueve Clinquot champaing will be sent to him shortly.

The author has provided the following explanation of the name Seccubus:

Seccubus is a mythical creature that helps security professionals analyze and report the results of, repeated, vulnerability scans. Like its distant cousins the Succubus and Incubus the Seccubus is also a creature of the night. At night, or any other scheduled time, the Seccubus draws its energy from repeatedly performing vulnerability scans  of infrastructures until the vulnerabilities become exhausted or die.
The Inseccubus is the male counterpart of the Seccubus. While the Inseccubus draws his life energy from the assessor by repeatedly requiring him to (re-)analyse the same findings, the Seccubus get her energy from pleasing the assessor by reducing the number of findings by means of delta reporting.

The name Seccubus was chosen from a list of over 50 ideas sent after the contest was announced via the AutoNessus.com website, Hacker Public Radio, Paul dot com and various other social media outlets like Twitter, Facebook and LinkedIn.

“I wanted a name that was completely different from AutoNessus” said Frank Breedijk, explaining why suggestions like AutoVAS and AutoVAMP where turned down. Other suggestions where turned down because their name was already taken on media like twitter (e.g. VAsak, Vulnerability Assessment Swiss Army Knife) or “simply because I didn’t like them” (e.g. Mick Douglass is awesome).

Now that the new name has been announced the “rebranding” will be complete before the end of the year. The website www.seccubus.com is already live but still points to the AutoNessus.com site. Also Frank’s twitter account, @autonessus, will be renamed to @seccubus soon.

The response to the renaming contest was overwhelming and we would like to thank everybody who participated.

More of often then not, the contractual agreements between assessor and client and client and service provider together with a “third party waivers” or similar documents do not cover everything that the three parties want to commonly agree upon. After reviewing quite a number of these documents, I decided to write a template agreement (which can be downloaded below) for exactly this situation. This document is not a replacement for the agreement between the client and the assessor, but as an additional agreement between all three parties.

Madison Gurkha and ITsec have both reviewed and contributed to this agreement and we will use it in our future dealings.

The agreement covers the following topics.

Scope of the assessment:

• What will be tested?
• When will the test take place?
• What kind of tests will be conducted?

Contractual agreements:

• Does the assessor have a contract with the client?
• Does the client have a contract with the service provider?

Legal liability:

• Do both the client and the service provider waive prosecution of the assessor?

Risks:

• Are all parties aware of and agree to the risks of a security assessment?

Practical matters:

• The client requests the service provider to support the assessment
• Who are the points of contact?
• Where will the assessment take place?
• How will the results be reported?

Confidentiality:

• All parties agree to confidentiality

The agreement template is released without any reservations of rights. This means you can use and adapt this agreement as you see fit, but completely at your own risk.

You can download the agreement here:

• Security Assessment Agreement Outsourcing v1.0 (Word document)
• Security Assessment Agreement Outsourcing v1.0 (PDF)

I would like to thank the following people for their contribution:

• Madison Gurkha: Hans van de Looy and Arjan de Vet
• ITsec: Tjerk Nan and Jan van Ek
• Fox-It: Mark Koek
• Arron Finnon (aka @f1nux)
• Colin McLean
• Robert Ladyman

This afternoon/evening, Security Justice will hold their 1st Annual International Podcast BBQ to celebrate US labor day.

The BBQ will feature our Schuberg Philis colleague Frank Breedijk as blogger for cupfighter.net and author of AutoNessus

At 15:00 EST (20:00 GMT) they will kick off by firing up the grill and opening the (probably not first) beers. After this there will be a series of interviews:

16:00 EST (21:00 GMT)  – Our own Frank Breedijk (@autonessus)
17:00 EST (22:00 GMT) – Chris John Riley (@ChrisJohnRiley) and Robin Wood (@digininja)
18:00 EST (23:00 GMT) – James Arlen (@myrcurial)
19:00 EST (00:00 GMT) – Nick Owen (@wikidsystems)
20:00 EST (01:00 GMT) – Clean-up and the usual banter…

The podcast will be streamed live via hak5radio.com and IRC: irc.freenode.net #securityjustice will be used for audience participation.

The .Net System.Version assembly offers a CompareTo method which can do the trick, as shown in the figure below.

The CompareTo method will return 1, 0 or -1  depending whether the compare to version  is higher, equal or lower.

Thanks to Shay Levi (see the comment) I now know a better/faster method for comparing version numbers (thaks Shay). PowerShell has its own [vesion] type. This removes the need of loading the assembly and using New-Object. It still allows for using the CompareTo method and direct compare via -ge, -gt, etc.

The CompareTo method will distinguish between the 3 possibilities (>, < or =), but direct comparison might be sufficient in a script.

Slowloris (name after the slow moving primates is a httpd DoS tool written by RSnake of ha.ckers. It works by tying up the httpd worker processes by slowly sending more and more headers of an httpd request.

Nkiller2 is a TCP/IP DoS attack tool which was published in issue 66 of Phrack magazine. It works by tying up httpd worker processes by requesting a file then stalling, mimicking the behavior of a client with full TCP/IP receive buffers.

Cisco CSS is a load balancer produced by Cisco.

In nearly all of the infrastructures built by my employer Schuberg Philis, the web servers are located behind a load balancer. In most cases a Cisco CSS. Because some of our customers were worried, I set out together with my colleague Gert Kremer to see if having a CSS load balancer in front of the web server provides any protection.

Slowloris

First we just had to try and find out what Slowloris did with an unprotected Apache server. The first video shows what happens when you run slowloris against a webserver. The window on the top left shows the number of apache processes, the top right window shows the scoreboard. This shows what the http processes are actually doing. The bottom window shows the slowloris output.

Slowloris vs Apache (No load balancer)

Click here to view the embedded video.

When slowloris is using 100 sockets, you can see 100 httpd workers in state “R”, meaning it is reading requests. The same is the case when running with 200 and 250 sockets. When running with 300 sockets the apache worker processes pool is exhausted and the web server can no longer service requests.

Slowloris vs Apache behind a Cisco CSS load balancer

Click here to view the embedded video.

Slowloris is running against the webserver with 3000 sockets (should be more then enough). As you can see on the top two windows the load balancer does not forward any of the incomplete requests to the webserver. We have stress tested the loadbancer up to 10,000 sockets and it had no effect on the loadbancer.

NKiller

Nkiller vs Apache (No load balancer)

Click here to view the embedded video.

In the video we see for windows. Top left and right show the number of apache processes and the apache dashboard. The middle window displays the NKiller output and the bottom window TCPdump.

When NKiller starts we see the it exhausts the httpd workers processes by putting them in a state where they are hanging while writing their reply back to the client.
Nkiller vs Apache behind a CSS load balancer

Click here to view the embedded video.

When NKiller was used against a server protected by a Cisco CSS load balancer the packets received from the load balancer do not match the expections of the Nkiller tool and the tool crashed producing a segmentation fault.

For example: scripts manipulating files will often use the Get-ChildItem cmdlet in combination with the -recurse parameter, but not allways the subfolder files are required. Instead off having multiple Get-ChildItem commandlines (each with their own set of parameters) a single line might be possible.

The following examples use the Get-ChildItem cmdlet to show what I mean.

The command in figure below will display all *.tmp files in the current folder and its subfolders.

By extending -recurse with :$false recursive lookup will be turned off.

On the otherhand, replacing $false with $true will turn recursive lookup on again.

So using a boolean variable we can turn recursive lookup on or off from within the script (-recurse:$RecurseOnOff). And of course this method also works for other parameters.

And it does get stranger….. in some occasions you can also reverse the default action of a parameter. Hence the following figure.

The 1st command will show all files with exception of the *.tmp files. By appending :$false to -exclude, we turn -exclude into -include as demonstrated by the 2nd command.

This form of parameter manipulation offers a scala of possibilities. Using script parameters to control cmdlet behaviour can both decrease script size and complexity.

Have fun experimenting with this little trick

Much has happened during the past year. A list of up till now mostly “orally transmitted” rules have been cast in policies. New procedures (e.g. the Assurer Challenge) and obligations (e.g. in the CAcert Community Agreement) have been decided. The Assurer Training Events try to bring all this informations to “the people”:
- To what, does the CCA protect every CAcert-Community-Member and as such also you?
- Can you recount the 5 statements of the “Purpose of Assurance”?
- Can you at least recount 10 security marks of the Dutch passport/Identity card?
Answers to these and following questions are given at the Assurer Training Events (ATE’s).
Participation in the events is free, Contributions are however appreciated.Amsterdam:
—————
The ATE-Amsterdam takes place on:
- Monday, June 15th from 20:00 till 22:00
- at SCHUBERG PHILIS
Star Parc
Boeing Avenue 271
1119 PD Schiphol-Rijk
—————
The ATE-Eemnes takes place on:
- Saturday, June 20th from 10:30 till 12:30, followed by normal assurances till 15:30
- in de Hilt
Hasselaarlaan 1c
3755 AV Eemnes
The Event-Team is already excited about your participation.
Registration ATE-Amsterdam
Registration ATE-Eemnes
contact: events@cacert.org

Our proposal has to be delivered to them before the weekend.  But we still have several hours of quality time ahead of us.

I’d love to share all the details, but erhm, since this is all hush-hush stuff I can’t.

But to give a general idea: DTAP hosting platform for enterprise application, internal use only, user population (>5,000 employees) located around the globe, highly confidential and secure, twin versus dual data center setup located in Amsterdam area, partially virtualized in network and server layers, internet as well as Corporate connectivity, authenticated against corporate user repositories, Windows based, maximum data loss of 1hr (RPO), 10TByte data, IDS/IPS, the lot.

And then suddenly find myself working > 2am. Geeh. Time to take a nap and kick ass again tomorrow.

Grtz,