# 20091218 – vo.o1 – PZO    – Initial hack to delete hardware profiles in Windows Vista/7
#
#————————————————————————————————————————————–
# Let’s see which profiles exist..
#————————————————————————————————————————————–
$i = 0
Write-Host “”
Write-Host “The following hardware profiles have been found on this computer:” -f white
foreach ($profile in (ls -path “HKLM:\SYSTEM\CurrentControlSet\Control\IDConfigDB\Hardware Profiles\”) ) {
Write-Host 000$i – (get-itemproperty -path “HKLM:\SYSTEM\CurrentControlSet\Control\IDConfigDB\Hardware Profiles00$i”).FriendlyName
$i++
}
#————————————————————————————————————————————–
# Now we can ask which to remove..
#————————————————————————————————————————————–
Write-Host “”
Write-Host “You are strongly advised not to remove profile 0000 – New Hardware Profile” -f red
Write-Host “”
$input = read-host “Which profile is causing you headaches and should be removed?”
Write-Host “deleting.. “HKLM:\SYSTEM\CurrentControlSet\Hardware Profiles\$input”"
Remove-Item -Path “HKLM:\SYSTEM\CurrentControlSet\Hardware Profiles\$input”

Finally: do _not_ remove profile 0000 unless you know what you are doing. YMMV!

[BBG]

While doing a performance test on one of our customer environments we observed the impact of TCP offload and “Receive Side Scaling” (RSS) settings on the interface card on Windows web servers in combination with traffic handling.

Setup:

1. 2x Mercury Load Runner generators hitting public URL of customer

2. Served by 3x Windows2003 SP2 servers, running IIS6

3. Load being balanced by Cisco CSS11503 to web farm.

The CPU performance graph of the web servers with TCP offload and RSS enabled on the internet facing (FRONT) interface:

Similarly but a more outdated graph even more clearly showing that traffic is alternating from one web server to another:

Most interesting right!?

What makes this traffic to alternate if the load balancer has been set up to distribute the load evenly across the farm resp each Load Runner vuser to clear its cookies and session cache after each request?

We then stumbled over this read, knowing that TCP offload to network card is a classic one , but still:
http://blogs.msdn.com/psssql/archive/2010/02/21/tcp-offloading-again.aspx

And found out the characteristic that when TCP offload and RSS were disabled, the load is more evenly spread across the web farm:

I find this pretty cool.

Any comments?

Find out how

To prepare for a specific monitoring user follow these steps:

Create a monitoring profile

Now create your monitor (in the authoring section).
See for basic instructions on how to create a monitor one of my previous posts: http://www.cupfighter.net/index.php/2009/10/check-your-sql-backup-automatically/

Here is a sample script with some basic options for passing the output to the eventlog and to SCOM itself to set the state of the monitor and generate alerts. The script also contains some code to determine the user account that is used.

Option Explicit
Dim checkdotcomma, strStatus
Dim objAPI, propertyBag
Dim objWMIService, colProcesses, objProcess
Dim strCurrentUser, User, Domain, strUserList

On Error Resume next
Const EVENT_TYPE_ERROR = 1
Const EVENT_TYPE_WARNING = 2
Const EVENT_TYPE_INFORMATION = 4

‘ Check if we are using the correct user for this check
Set objWMIService = GetObject(”winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2″)
Set colProcesses = objWMIService.ExecQuery(”select * from win32_process where Name=’cscript.exe’”)
For Each objProcess in colProcesses
If objProcess.GetOwner (User, Domain ) = 0 Then
strCurrentUser = “Script has run under account: ” & Domain & “\” & User
Else
strCurrentUser = “Problem getting the owner for process ” & objProcess.Caption
End If
strUserList = strUserList & strCurrentUser
Next

Set objAPI = CreateObject(”MOM.ScriptAPI”)
‘ perform check on regional settings if numbers are using dots or commas
‘ replace this with your own code you want to run
checkdotcomma = Mid(1/2,2,1)
If checkdotcomma = “.” Then
strStatus = “Ok”
Call objAPI.LogScriptEvent(”CheckDotComma”,2000, EVENT_TYPE_INFORMATION,”Regional Settings are using a Dot (.). The user list is ” & strUserList )
Else
strStatus = “Error”
Call objAPI.LogScriptEvent(”CheckDotComma”,2001, EVENT_TYPE_ERROR, “Regional Settings are using a Comma (,). The user list is ” & strUserList )
End if

‘ return status to monitor
Set propertyBag = objAPI.CreatePropertyBag ()
Call propertyBag.AddValue (”Status”, strStatus)
Call propertyBag.AddValue (”checkdotcomma”, checkdotcomma)
Call objAPI.Return(propertyBag)

Download the System Center Operations Manager 2007 Authoring Console

http://www.microsoft.com/downloads/details.aspx?FamilyID=6c8911c3-c495-4a03-96df-9731c37aa6d7&displaylang=en

To make it bit more nice, export your management pack, and look up the Secure References.
Replace all instances of the SecureReference ID with a more readable format, see below.

<SecureReferences>
      <SecureReference ID="MonitoringUser" Accessibility="Internal" Context="System!System.Entity" />
    </SecureReferences>


Reimport your managementpack and you are all set.

So with all this power and cooling hardware in place, we’re protected against everything… right? Well think again, because the power grid and air conditioning systems are also controlled by…. software! A seemingly harmless software update to the ACU’s inside one of our suites caused a control valve to react in the opposite way its control software thought it was sending them, effectively shutting down cooling and causing a 10 degrees centigrade temperature rise in little over 30 minutes. These are the type of temperature rises which ultimately cause hardware to auto shutdown. In this case, the problem was cleared before reaching critical levels. If it hadn’t, we would have been able to transparently fail everything over to a remote location, since the typical infrastructures we build are based on a twin data center active / active concept.

This again proves that it doesn’t always have to be the often cited ‘plane crash’ which proves the point for building mission critical infrastructures, like our customer’s, inside multiple data centers. Actually, I don’t think there are any recorded events of an airplane crashing into a data center. Instead, something like the firmware controlling your ACU’s can jeopardize all equipment inside a single room or even an entire data center. Plan for failure and expect failure to come from unexpected sources.

All things considered, the twin datacenter active/active configuration is indeed too hot to handle!

I figured to remove and reinstall, keeping the database and logs, but every reïnstall kept failing and bombing at about 90% out with a dialogue box stating ‘there is something wrong with your installation package’. As I knew for sure the package was fine (I did try both the SP1 and SP2 install..) it must be something else.

The logfile MWusSetup.log located in the Windows temp folder mentioned: ERROR CustomActions.Dll  RemovePsfsip: Failed to load dll  (Error 0×8007007E: The specified module could not be found.)

After a little googling, I found a lot of references, but not one fully working solution.

What worked for me is this (reboot after every step):

Removed all dotnet installs using a MS utility cleanup_tool.exe
(http://blogs.msdn.com/astebner/attachment/8904493.ashx)

Stop and remove the WsusCertService using the 2003 resource kit utility instsrv.exe
(http://www.microsoft.com/downloads/details.aspx?FamilyID=9D467A69-57FF-4AE7-96EE-B18C4790CFFD&displaylang=en)

Cleaned the registry using ccleaner.
(http://www.ccleaner.com)

Reïnstalled .Net3.5SP1
(http://download.microsoft.com/download/2/0/e/20e90413-712f-438c-988e-fdaa79a8ac3d/dotnetfx35.exe)

Removed the wsus mmc cache files in my profile directory.

This finally allowed me to reïnstall WSUS.

[BBG]

Recently we started evaluating Citrix Edgesight, on a enviroment we are currently building, consisting of XenApp5 2008 x64 and XenDesktop 4 Farms.

After the installation of the EdgeSight agent, suddenly a bunch of applications running within a Java Virtual machine stopped functioning. Throwing the “Could not launch the java virtual machine” error.
These Java apps tried allocating quite some memory using these java arguments (eg: XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=35 -XX:NewRatio=2″   initial-heap-size=”32m” max-heap-size=”1024m”)

After some investigation a colleague (Hugo Trippaers) found out that there was only 0,9 GB of memory allocatable on our Citrix XenApp machines using the memtest32.exe tool. While our other servers happily reported 1,5 GB of allocatable memory (Within WOW64). (Physical Machine = HP DL380G6 with 48 GB of memory, uh should be enough?)

After some deeper digging using memalloc.exe, I discover some substantial differences in memory allocation between our XenApp Servers with the edgesight agent installed and servers without the EdgeSight agent.

XenApp servers with Edgesight Agent 5.2 SP1 x64: memalloc.exe with edgesight
XenApp Servers without edgesight: memalloc.exe – without edgesight

The main difference here is all the Citrix hooks being loaded, see below.
This apparently consumes so much memory that it was not possible for java to allocate enough memory.

For more insights on WOW64 look here:  http://blogs.msdn.com/gauravseth/archive/2006/04/26/583963.aspx

By default 32bit applications within WOW64 can leverage the full 4 GB of memory availlable, which is not possible on a native 32 bit system because of the separation of kernel and user space.
Applications need to be compiled with /largaddressaware (Visual Studio : http://msdn.microsoft.com/en-us/library/wz223b1z(VS.80).aspx) or patched using editbin (http://bilbroblog.com/wow64/hidden-secrets-of-w0w64-ndash-large-address-space/), to fully use the 4 GB availlable otherwise they can only allocate 1,6 GB of memory.

We will open a case with Citrix on this; to be continued.

Citrix hooks being loaded when edgesight is installed:

Address 61200000, length 1000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\EdgeSight\Agent\Agent\Core\rsintcor32.dll
Address 61201000, length 18000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\EdgeSight\Agent\Agent\Core\rsintcor32.dll
Address 61219000, length 9000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\EdgeSight\Agent\Agent\Core\rsintcor32.dll
Address 61222000, length 1000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\EdgeSight\Agent\Agent\Core\rsintcor32.dll
Address 61223000, length 4000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\EdgeSight\Agent\Agent\Core\rsintcor32.dll
Address 61300000, length 1000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\edgesight\agent\agent\edgesight\esint32.dll
Address 61301000, length 8000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\edgesight\agent\agent\edgesight\esint32.dll
Address 61309000, length 3000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\edgesight\agent\agent\edgesight\esint32.dll
Address 6130c000, length 1000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\edgesight\agent\agent\edgesight\esint32.dll
Address 6130d000, length 2000, state 1000 MEM_COMMIT, type 1000000 module e:\program files\edgesight\agent\agent\edgesight\esint32.dll
Address 67f60000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\CtxSbxHook.DLL
Address 67f61000, length 58000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\CtxSbxHook.DLL
Address 67fb9000, length a000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\CtxSbxHook.DLL
Address 67fc3000, length 4000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\CtxSbxHook.DLL
Address 67fc7000, length 7000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\CtxSbxHook.DLL
Address 6db20000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 6db21000, length 96000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 6dbb7000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 6dbb8000, length 2000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 6dbba000, length 4000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 6dbbe000, length 5000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_5090ab56bcba71c2\MSVCR90.dll
Address 751e0000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 751e1000, length c6000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752a7000, length 3000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752aa000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752ab000, length e000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752b9000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752ba000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752bb000, length 6000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 752c1000, length 5000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\twnhook.dll
Address 75320000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 75321000, length 63000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 75384000, length 2b000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753af000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753b0000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753b1000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753b2000, length 1000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753b3000, length 3000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753b6000, length 5000, state 1000 MEM_COMMIT, type 1000000 module C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4016_none_d0893820442e7fe4\MSVCR80.dll
Address 753c0000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753c1000, length 1d000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753de000, length 4000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753e2000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753e3000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753e4000, length 3000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\scardhook.dll
Address 753f0000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\wdmaudhook.dll
Address 753f1000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\wdmaudhook.dll
Address 753f3000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\wdmaudhook.dll
Address 753f4000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\wdmaudhook.dll
Address 753f5000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\wdmaudhook.dll
Address 75400000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\System32\cxinjime.dll
Address 75401000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\System32\cxinjime.dll
Address 75402000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\System32\cxinjime.dll
Address 75403000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\System32\cxinjime.dll
Address 75404000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\System32\cxinjime.dll
Address 75420000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75421000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75423000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75424000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75425000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75426000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\Sfrhook.dll
Address 75430000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mmhook.dll
Address 75431000, length f000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mmhook.dll
Address 75440000, length 2000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mmhook.dll
Address 75442000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mmhook.dll
Address 75443000, length 3000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mmhook.dll
Address 75450000, length 1000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mfaphook.dll
Address 75451000, length 2c000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mfaphook.dll
Address 7547d000, length 9000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mfaphook.dll
Address 75486000, length 4000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mfaphook.dll
Address 7548a000, length 4000, state 1000 MEM_COMMIT, type 1000000 module E:\program files\Citrix\system32\mfaphook.dll

So basically DARPA hid 10 red weather balloons all over the continental United States, and the challenge was to find them all, submit their latitude and longitude, and to find them first. Of course a team from MIT won the competition. How long did it take to find them? A month? A week? Just 8 hours and 52 minutes. How did they do this? By using social media and social networks of course.

Officially the DARPA Network Challenge states:

The DARPA Network Challenge is a competition that will explore the roles the Internet and social networking play in the timely communication, wide-area team building, trust and urgent mobilization required to solve broad-scope, time-critical problems.

So that’s all well and good, fun and interesting and such. But the thing that got me thinking, the thing touched on in the marketing website article was not the discovery of the (in advertising lingo) “big idea” a.k.a. the red balloons. But rather it was the MIT team’s process and approach to solving the problem that is the new “big idea.” The process invented by MIT’s team to rapidly assemble and task it’s newly formed “red balloon team” community worked, and it easily slipped into the operational ethos of bloggers, Facebook users and Twitter users (of course, having decided to donate the $40,000 cash prize to a charity probably helped too). The success of that process demonstrates to me (and DARPA who will interview the MIT team and it’s “community” of participants) the real value of social networks and the internet.

What the marketing website article is trying to say is that ad agencies used to be doing nothing but looking for the next “big idea” and then pitching it to their clients. But along came the internet and changed all that. There are plenty of these big ideas to go around, and depending on how immersed you are in all this social media/networking stuff, more and more of them are starting to come from end-users or consumers. Take the Swiffer for example, it was an idea suggested by a consumer responding to an initiative called “Connect and Develop” from Proctor and Gamble to gather feedback and ideas from their customers.

Crowd sourcing: No one is as smart as everyone.

This is one of the ideas that forms the center of the disruptive technology called the internet. We experience successive waves of change that are emanating from the fact that virtually anyone can publish their thoughts, ideas, images, and video for the rest of the world to find. And sometimes conditions conspire to allow a simple idea or thought to permeate the minds and hearts of millions of people in a near instant. Such things are often called internet memes.

The first wave that hits you is email. Everyone starts here and sees the value of being able to send and receive email. Even my parents have been hit by the power of this medium of communication. The next wave I think that hit was port 80 traffic: http protocols for websites and web pages. Then e-commerce as a wave of online shopping, followed by an MP3 wave (napster at first, iTunes music store now), and most recently by a youtube.com or video wave.

In each of these waves, traditional media entities have been deeply disrupted by the free flowing of ideas and assets. Email killed the telegram (Western Union decommissioned the service in 2006 after over 150 years of use) and is digging into postal service revenues since day one. The websites and webpages have largely up-ended magazines and newspapers so that printed editions are now becoming increasingly scarce. MP3s have both salvaged and savaged the recording industry. And in January 2009 YouTube.com recorded over 100,000,000 viewings per day.

So all of this will continue happening, the waves of disruption (disruptive to traditional thinking and doing at least) will keep on coming. Publishing will become easier, in all sorts of media. Access will be expanded to include more and more people. And our part in all of it, at least in my view, is to remember to try to step back and think about the process of change that is going on. The new ways we can solve problems using this incredible web of technologies and people addicted to them. That will remain a valuable skill and insight to achieve and maintain. Learning how to program perl is great, or some other language. But eventually perl won’t matter that much. We won’t need to pay so much attention to the underlying technologies of the internet because they will (rightly) recede into the background. What will remain will be pure freedom of communication and expression I imagine. And the possibilities at that point will be blinding. So don’t fret about the big red balloons, just try to keep being a curious, problem-solving clever monkey and you’ll always have interesting work to do.

Since the root CA is in another domain then the issuing CA, it took some fiddling and tweaking around with my CDP and AIA extensions, but that is another blogpost all together.

I knew I was in for some fun when when the following happened:

• I installed my Issuing CA and generated the certificate request
• I issued the request to my Root CA and generated the Issuing CA certificate
• I tried to install the Issuing CA certificate and got the following error:

Cannot verify certificate chain. Do you whish to ignore the error and continue? The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2168885613)

My first reaction was to call one of the network guest and notify him that I needed http access to the Issuing CA to the CDP location. But whil on the phone, I decided to try and to my surprise I was actually able to manually pull down the crl.

Intregued, I decided to check a few things:

• I could download the CRL from both CDP locations with Internet Exporer
• I could open the downloaded CRLs
• I could telnet to port 80 of the both webservers
• I could telnet to port 80 manually issue the GET /crl/CRLname.crl HTTP/1.0 command and get data back

O.K. what is going on here… Lets open PKI view, which is now included in Windows 2008 and Vista and can be downloaded for Windows 2000 and 2003.

It seemed that PKI view as in agreement, it too could not download the CRL from the CDP location

PKI view shows "Unable To Download" for both CDP locations

This did sent me on a wild goose chase:

• Microsoft own documentation, clearly blames it on unavailability of the CDP location, something I, by now, had triplle checked four times and refused to believe
• This “Network Builders” forum and many others, simply suggest to turn off revocation checking, but that is clearly not a worthy solution either.
• Apparently there is also an issue with serving delta CRLs threw IIS because the + sign at the end of the basename of a delta CRL file leads to so called “double escaping”. I could rule this out by looking at the IIS logs.
• In the end this technet forum post, about OCSP reponders Brian Komar points out:

But, as stated, I would use certutil to get the “best” answer on how is my configuration.
Certutil -verify -urlfetch “certfile.cer” will check *every* CDP and AIA URL (including OCSP) and tell you how they are all doing *at that specific instance in time” since it goes to the URLs immediately.
Brian

I exported the Issuing CA certificate from the certificate database of the Root CA and ran the command against is and this is what I found

E:\>certutil -verify -urlfetch <certfile>.cer
Issuer:
CN=Root CA
Subject:
CN=Issuing CA
Cert Serial Number: 115d5f6400020000000b
<snip>

—————-  Certificate AIA  —————-
Verified “Certificate (0)” Time: 0
[0.0] http://IIS1.domain1local/crl/Root-CA.crt

Verified “Certificate (0)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crt

—————-  Certificate CDP  —————-
Wrong Issuer “Base CRL (13)” Time: 0
[0.0] http://IIS1.domain1.local/crl/Root-CA.crl

Wrong Issuer “Base CRL (13)” Time: 0
[1.0] http://IIS2.domain1.local/crl/Root-CA.crl

<snip>
E:\>

So while PKI view and the other error messages I was getting all pointed to the most common cause, it actually turned out that the CRl did get downloaded, but was not cryptographically relevant to what the system believes is the Root CA certificate.

Root cause

Inspection of the CRLs generated and the Root certificates installed showed what had caused the problem. In order to test the CDP extensions I had reissued the Root CA certificate, causing the Root CA to have three active certificates. Each with a different key.

This CA has three CA certificates

When validating the Issuing CA certificate, validation would end at the last certificate issued, however the CA still signs its CRLs with the key pair of the first certificate.

I guess for me there is nothing left but to reinstall the entire chain.

The new Seccubus logo

Last month our coworker Frank Breedijk rechristened his vulnerability management tool Seccubus. Today he has launched his new website Seccubus.com

With the new website author Frank also unveiled the new logo for Seccubus drawn bij Schuberg Philis collegue Robert Heuvel.

I want to mention some training sessions I attanted

• Dtrace course by Jim Mauro and a lot of extra information came from Richard Elling and 1 other Sun employee. Together they provided a lot of real world examples on how to use Dtrace. And nice details about how it works in the kernel. Everyone knows Dtrace from the youtube movie by Brendan Gregg more info on his blog. So now I should enable all Dtrace probes and start screaming in the datacentre and see if I was loud enough

  Click here to view the embedded video.

• ZFS by Richard Elling, I never had time to look into this FileSystem before, so a great way to learn all about it in one day. One of the nice features is the buffering of disk-writes which gives a kind of breathing or heartbeat towards the disks. And with ZFS you can buffer writes to a solid-state drive before sending it to the “slower” disks.

• Jquery given by Tobias Oetiker,an easy way to build spiffy webpages that look the same on each browser. Like this demo . Got a really great explanation about the problem with the scope of variables in Javascript especcially because JQeury uses the “$” as a variable and how to get around it using a function. And there is a nice page with a lot of Jquery plugin material http://plugins.jquery.com.

• Nagios Advanced Topics by Sellens , I discovered that the feature I am still missing in Nagios isn’t build yet , having two nagios hosts loadbalance the load and keeping each other in sync. We already build our own solution of nagios hosts keeping eachother in sync only the loadbalancing part needs some work maybe I need to spend some time on reading the nagios mailinglist.

The Sun guys were really pushing or should I say selling opensolaris , well they were giving away a lot of opensolaris dvd’s and they mentioned the website http://www.solarisinternals.com/ a lot. Really cool to see all the buzz about an open system.
In the hotel I had breakfast with Mike Ciavarella, we spoke about his training session about documentation and how it would secure your job and even helps getting a better position.

Attended a lot of BOF Session , one of them was with D.Brent Chapman from Netomata. About the automation of network Configuration and Management it brought back a lot of memories of the times I was managing systems that configure and monitor ADSL modems. People just turn of their modem and I needed to figure out if this was an outage or a Human action, that was fun.

Sjoerd already mentioned the national democratic institute, what really stayed in my mind is that everybody is trying to encrypt as much as possible, and think about social engineering to get information. The people at ndi need to work different, they make sure never to encrypt stuff and be as open to the world to get their Institute accepted by getting trust from governments and groups in the difficult areas where they work. Every time when I use GPG to keep others from reading my data I think about the guy we met at Lisa09.

During a Google-Wave sponsored drink met some people from Research in Motion (RIM) that manage the linux servers that make all connections from the RIM towards google , msn etc possible.

So had a lot of fun at #lisa09 , and nice weather too.

So LISA09 took place between 1 and 6th of November, 2009 in lovely Baltimore, MD. I chose to follow more the tutorials (trainings) path. Got five tutorials – one bad, two medium and two nice ones. The problem with tutorials is that sometimes they are very basic which I really didn’t expect to be a case on such event.

So a quick summary on the thing that got my attention.

A non-technical tutorial about management skills. Looked like regular project/team management training but with special attention to the idea of geeks being managed by geeks. So no suits in the area. As we all know it is not easy to manage highly skilled individuals or being a “proxy” between business and the geeks. One of the big issues – managing people way older and more experienced than you are – was also covered. Unfortunately it was just confirmed to be not that easy to do..

Lot of noise around ZFS. Nice training about the file system internals. Also other trainings given by James Mauro and Richard Elling were very interesting – DTrace, Solaris 10 performance tuning etc. SUN (probably as always at LISA) was very noticeable at the conference. Evening sessions kind of Schuberg Philis’ style although they still need some practice in that area

cfengnine3 vs. puppet – unfortunately didn’t attend to any of those workshops but you could smell a gun powder in the air

XEN – as great it is everyone knows but they really should spend some effort in doing a good presentation/training because the one which took place in Baltimore wasn’t the best way to get on board with it.

Google was giving away a free stuff, it was very popular..

Over the Edge System Administration – nice one, basically non standard system administration procedures (behavior). Stuff that happens for everyone from time to time. Do something not very common which works and sometimes stays in production for years.. Kind of BOFHish too..

Maybe some tips for people who will attend in LISA in future:

• Avoid trainings unless you’re really sure that level/scope will really suits you
• Wear black clothes – people will like you more
• Grow a beard – you will be respected more

From what I understood from people who had been here previously, the attendee list was a lot smaller than previous years. But still, there were more than enough people to share a talk with. It was good to have the opportunity to talk to people working at some big and very known companies like Yahoo, Pixar etc. But also I met some people who worked for less know companies (at least for me) but maybe even more interesting companies, for example, the national democratic institute.  A non-profit organization facilitating democracy in countries where democracy isn’t that natural as in most western countries. I don’t think a lot of system admins have to worry about problems like militia stealing servers from your datacenter.

The first 5 days I followed a set of trainings, some days training for the whole day, some days a morning and an afternoon session. In general I was a bit disappointed by the trainings, they covered a lot of basic stuff, a whole day can be a very long sit for just 2 new bits of information. But a few sessions were quite interesting and/or entertaining.

One of the more interesting ones was the IPv6 talk by Rudi van Drunen (who will also be the program chair for next year). The information wasn’t particularly new or interesting (‘We really need IPv6 because we will be out of IP’s very soon!’) but it’s funny to listen to a fellow Dutch guy giving a talk on the other side of the ocean. Makes you feel a bit at home

Another talk which I found interesting was the Time Management for System Administrators by Thomas A. Limoncelli, a nice and funny guy to listen too with very practical tricks and solutions to spend your time more efficiently. Especially his tips on spending as less as possible of your time in meetings were very good. One of the more funny one was, tell the other people that you really have just a half hour of time, and ask them to move your subject to the start of the meeting and leave after half an hour. Not really nice, but efficient.  One I really liked was, to let everybody write down the items they want to discuss in a wiki with a death line of one day before the meeting and cancel the meeting if no subject were brought up. This ensures that meetings are only organized on a need to have basis.

On the Friday and last day I planned a Tech Sessions Day, this has more the setup of a standard conference, 3 tracks of various topics all about 2 hours each. I enjoyed this setup a lot more. It’s probably better suited for my short attention span . I especially liked the talk from 2 Google guys, who were presenting some software they build while working for Google, and were now releasing it to the public. It gave me some insight in how Google is developing software and approaching system administrative problems. The software is a kind of meta language enabling a generic way of managing the configurations of various firewall equipment (Cisco, Juniper, Iptables).

All together it was a long week, but well spent with a lot of new experiences and also a nice opportunity to learn a bit more about my co-workers.

Raoul is a member of UNICRI (http://www.unicri.it/), a United Nations crime and justice research institute.

Unicri research technology as well, because if normal people use technology, the bad guys use it as well.

“Every new technology opens the door to new criminal approaches”

In the 70s the first wave of hackers where searching for knowledge. In the early 80s the second wave of hackers was driven by curiosity. The third wave of hackers in the 90s where eager to hack and started to exchange information. The first communities where created. The current fourth wave is now driven by anger and money. Hacking has met politics (hacktivism) and money (cybercrime).

Why is cybercrime on the rise?
1)    There are more and more targets, thanks to broadband
2)    A need to make money, think economical crisis
3)    Hacking got easier, 0-day attacks and skimmers can be easily bought online.
4)    Fall guys are easy to recruit, e.g. for money laundering
5)    The criminals think they cannot be caught
6)    There is no violence, no need to face your victims

Hackers are no longer part of the ICT community, they are in it for the money and are professionals, but the media image of ciber criminals is still the old hacker image. Sometimes today the hackers are the good guys and the professionals are the bad guys.

Some numbers on cybercrime:
•    285 millions records compromised in 2008
•    $2,000,000,000 yearly turnover of RBN
•    148% increasing in ATM fraud

RBN is the Russian Business Network, its basically the ISP for cybercrime, they offer hosting and good bandwidth to those running a criminal enterprise on the web. It will give user anonymity and interaction with “like souls”.

Underground economy is the mechanism to clean money. Stealing money is easy, using that money is not so. Therefore any cybercriminal needs to set up a money laundering operation.

It is an organized enterprise.
Hackers, coders and scammers get the money for the boss and the mules make the money clean for him. Underground economy is everything from trading stolen information and good, the services needed to get them and the services to clean the money.

So how does this economy work?

In order to trade goods (CCV codes, cards, credentials, identities) on online forums you have to be approved by the organization that runs the forum
Fake credit cards are of  high quality and look very legitimate?
On line checks give full details of card holders for card production.

www.darkmarket.ws was run by two business mans. ChaO was arrested, but has been for years one of the biggest sellers of ATM skimmers. His villa, with personal swimming pool, contained a hologram printer, 10 boxes of skimmers and lots of fake cards.

These guys live in luxury.

Tor is really a community of developers and volunteers and is still looking for developers and volunteers to enhance themselves.

Top countries in the world in bandwidth:
•    Germany
•    USA
•    Netherlands
•    France
•    Sweden

Anonymity means different things to different people:
•    Private citizens – Privacy
•    Government – Traffic analysis resistance
•    Human rights activists – Reachability
•    Businesses – Network Security

Tor gives three anonymity properties by design, nto by policy:
1)    A local network can learn of influence your destination
2)    No single router can link you to your destination
3)    The destination or somebody watching it cannot learn you location

Tor is constantly being attacked, not by attacking the code, but by:
•    Blocking the directory authorities
•    Blocking relay IP addresses in the directory
•    Filtering based on Tor’s fingerprint
•    By preventing users from finding the tor software

Outers/IPS-es could filter on Tor’s signature in the past, but it now looks like Firefox talking to Apache. When the Tor download website was blocked, the Tor project test up a download tor by email service.

When the Peoples Republic of China turned 60 years, the censorship stepped up in preparation for it. Protecting the torproject.org website with an SSL certificate was good enough in the pas. They also took a snapshot of the network and blocked all its ip addresses for the day of the anniversary. Jacob showed a graph that showed us what suppression looked like.

As a reaction users where able to still get on the Tor network via bridge which you could get via email, or that is kept private.

There is quite a bit of censorship going on in the Western world, this is not something exclusively for evil regimes.

If you want to help the Tor project go to http://torproject.org and download and install the software.

The talk is about the “Smart Grid”. The key components are and advanced metering infrastructure, Transmission and distribution and generation of electricity.

Advanced Metering Infrastructure enables two way communication between the meters in your home and the power company. It offers the following features:
•    Load control works like this: Some power offer a discount in return for control over the thermostat of your AC or by allowing them to turn off your clothes dryer during peak hours. The main reason for this is officially to prevent black outs, but it can be used to prevent penalties as well.
•    Demand response: It allows for dynamic rates to be loaded to your meter.

Why move to a smart gird?
•    Energy conservation
•    Cost reduction
•    Improved Reliability of Delivery

Smart Grid security is significant because it has national security implications, because there are millions of entry points into the grid.

Why attack a smart gird?
•    Financial gain
o    Hacking your meter
o    Monitor the power usage for breaking and entering
•    Mischief
o    Turn off your neighbors’ power
•    Chaos

What are the attack vectors?
Meters are outside the houses in the US and their physical security depends on a normal screw. This means you can do hardware reverse engineering.
Keying millions of meters is hard, so it is likely that keys predicatable.
Once the channel is hacked open, the fun can begin.

Meter talk to devices in the home using ZigBee, but it is broken. KillerBee, written by Josh Wright, allows you to do anything you ever wanted to do on a wireless project on a ZigBee network.

As more and more control hardware starts using wireless protocols like WiMax, Wifi or ZigBee the bigger the attack surface. Even if these things control systems that result in death or personal injury if they fail.

Almost no hardware has firmware signing.

A self propagating worm for smart power meters has been demonstrated by Mike Davis of ioActive with a payload that can change rates, brick a meter alter usage, etc.

There is already so much hardware deployed with defunct security that it is very hard to get it fix.

He also released, into open source, an offline MiFare cracking utility that can be used to crack any MiFare card for 30 euros and with just a few hours of work.

In the past MiFare’s encryption technology, Crypto1, was only available in hardware and thus survived for a surprisingly long time.

Pavol explained how his program can computer derived keys from the main key by using the time distance between the keys.

For those people that dodn’t know. MiFare Classic can be cloned in 99.6% (Except for sector 0 that cannot be written) a ProxMark3 card emulator can emulate all cards 100% perfect.

There are currently three countermeasures:
1)    User safe cards (Mifare Plus/Mifare Desfire or other)
2)    Use decrement counter protection (workaround)
3)    Use online checking

Slovak public transport card allows anybody to read the name of the passenger and has no protection against cloning or modification.

The tool can be downloaded from https://www.nethemba.com/research/ .

Coming up:
•    Cracking hitag rfid
•    Cracking GSM encryption

You can download the slides below:

• TLS renegotiation authentication GAP v1.1 pdf
• TLS renegotiation authentication GAP v1.1 pptx

Special thanks to Marsh Ray for his suggestions and corrections.

Slide deck “Seccubus Confidence 2009.02 v0.1″

On the 19th of November Frank Breedijk announced that Jason Mansfield, who runs the website http:/clinicallyawasome.com, has won the contest by sending in the name Seccubus. A bottle of Vueve Clinquot champaing will be sent to him shortly.

The author has provided the following explanation of the name Seccubus:

Seccubus is a mythical creature that helps security professionals analyze and report the results of, repeated, vulnerability scans. Like its distant cousins the Succubus and Incubus the Seccubus is also a creature of the night. At night, or any other scheduled time, the Seccubus draws its energy from repeatedly performing vulnerability scans  of infrastructures until the vulnerabilities become exhausted or die.
The Inseccubus is the male counterpart of the Seccubus. While the Inseccubus draws his life energy from the assessor by repeatedly requiring him to (re-)analyse the same findings, the Seccubus get her energy from pleasing the assessor by reducing the number of findings by means of delta reporting.

The name Seccubus was chosen from a list of over 50 ideas sent after the contest was announced via the AutoNessus.com website, Hacker Public Radio, Paul dot com and various other social media outlets like Twitter, Facebook and LinkedIn.

“I wanted a name that was completely different from AutoNessus” said Frank Breedijk, explaining why suggestions like AutoVAS and AutoVAMP where turned down. Other suggestions where turned down because their name was already taken on media like twitter (e.g. VAsak, Vulnerability Assessment Swiss Army Knife) or “simply because I didn’t like them” (e.g. Mick Douglass is awesome).

Now that the new name has been announced the “rebranding” will be complete before the end of the year. The website www.seccubus.com is already live but still points to the AutoNessus.com site. Also Frank’s twitter account, @autonessus, will be renamed to @seccubus soon.

The response to the renaming contest was overwhelming and we would like to thank everybody who participated.

The first part of his presentation explained why routers are interesting targets (they are in the core), but also why routers are not actually exploited that much. One of the reasons is that the attack surface of router is quite small because routers don’t expose that much services to a truly remote attacker and are rarely used as clients.

The exception to the rule is “cisco-sa-20070124-crafted-ip-option” which is a remotely exploitable bug that causes a stack overflow on the router. Since “nobody ever updates router software” this vulnerability is still very much alive.

But routers need to support more and more, like IPv6, VoIP, XML configuration interface, luckily most services off.

Writing exploits for Cisco IOS is hard because it is not a real OS, but a single ELF binary. It is not based on a real OS we know hoe to exploit. Its only option to recover from a critical fault is a full reboot.

Another thing that makes exploitation hard is the memory layout. It is different from each single IOS version that it out there, and there are quite a few, currently there are over 270,000 different IOS images known by Cisco and you cannot get the version number remotely.

Best bet for getting a reliable return address for router exploitation is Rommon, the routers bios which loads the IOS and then remains in memory. It is at a fix address and there are big pools of the same versions present on the internet.

Unlike his talk at BlackHat Felix actually showed how the crafted ip option exploit can be used to get working reliable exploit. But since IOS is not an OS you need to get away with it without killing the router. If the stack is not completely overwritten, the return registers remain in tack and thus can be used to reliably return. His method has one drawback, in order for it to work, you need to know the version, but it is not remotely identifiable.

As an alternative there are code similarities in IOS images, but this still has problems.

Felix also made progress on shell code, he showed code that would cause the password evaluation function to always return true.

How do you protect your router?
•    Have faith.
•    Don’t allow people to talk to your router
•    Protect your routing protocols
•    Don’t run services on routers
•    Treat your service cards as the linux machines they are

Running Rancid helps, modification of the data structures show up here.

Turn crash dumping on, this will make sure you keep evidence of any attacks.