Cupfighter.net

So you have performance troubles on Windows, you probably already pulled the sysinternals from the shelve. But did you already know the Windows Performance toolkit for hardcore performance troubleshooting?

This toolkit has three tools;

xperf.exe – Captures traces, post-processes them for use on any machine, and supports command-line (action-based) trace analysis.

xperfview.exe – Visual Trace Analysis Tool – Presents trace content in the form of interactive graphs and summary tables

xbootmgr.exe – Automates on/off state transitions and captures traces during these transitions.

So what do these tools do?

Performance Analyzer is built on top of the Event Tracing for Windows (ETW) infrastructure. ETW enables Windows and applications to efficiently generate events, which can be enabled and disabled at any time without requiring system or process restarts. ETW collects requested kernel events and saves them to one or more files referred to as “trace files” or “traces.” These kernel events provide extensive details about the operation of the system. Some of the most important and useful kernel events available for capture and analysis are context switches, interrupts, deferred procedure calls, process and thread creation and destruction, disk I/Os, hard faults, processor P-State transitions, and registry operations, though there are many others.

One of the great features of ETW, supported in WPT, is the support of symbol decoding, sample profiling, and capture of call stacks on kernel events. These features provide very rich and detailed views into the system operation. WPT also supports automated perf testing. Specifically, xperf is designed for scripting from the command line and can be employed in automated performance gating infrastructures (it is the core of Windows PerfGates). xperf can also dump the trace data to an ANSI text file, which allows you to write your own trace processing tools that can look for performance problems and regressions from previous tests.

More info:

http://blogs.msdn.com/ntdebugging/archive/2008/04/03/windows-performance-toolkit-xperf.aspx
http://msdn.microsoft.com/en-us/performance/cc825801.aspx
http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T594_WH08.pptx

Download the tools here:

• WPT Kit v.4.1.1 x86 version (4.4MB)
• WPT Kit v.4.1.1 x64 version (4.8MB)
• WPT Kit v.4.1.1 ia64 version (7.0MB)

So what is configuration management on Windows?

Let’s have a look at the average life-cycle of Windows Servers.
It all starts with the bare-metal install of the machines, most likely from a standardized image. All machines deployed with this image have basically the same configuration. Except for some variables like IP’s, SID’s and Hostnames. Let’s assume that in this example we deploy application X over a farm of machines manually. The installer of application X has all kinds of options / settings to define. When doing this manually it’s already possible that these machines get different configurations. To make things worse application X suffers from an issue on a single machine. So we start to troubleshoot the application and fiddle around with settings, maybe at some point we decided to call support and change even more low level settings and bring out the the debugging tools, network sniffers etc. Now configurations are even more different then machines running this same application. And over time things get worse and worse.

In the ideal world we want the systems hosting application X to be exactly the same, except from the variables which identify a specific system, like IP’s, SIDS’s and hostnames. When this is not the case, that we are at least aware of it.

Why do we need consistent configurations? Because it makes machines more predictable especially with the DTAP model commonly used within our company.

For example: if our A and P environments differ, configuration wise, it might be that you run into issues when propagating changes.

So basically we need a system which is able to bare-metal installs and deploy software in a standard manner without too much human intervention. But also a tool through which we can change configurations in a standard manner and/or a tool which can detect changes when done manually on a system (for example changing settings via a GUI) and/or force us to make the changes via this tool instead of doing this manually (Like CFengine). Basically a tool which (or tools) to support the lifecycle of an average windows server.

Where are these configurations?

If windows were like Unix with all configurations stored in text files and had somehow the same file system security as most Unix file systems, the choice would be obvious and we would take advantage of the experience with CfEngine. But when comparing Microsoft Windows to the average unix machine on the aspect of options to store configuration information, we soon discover that these operating systems are quite different.

Let’s see where configurations are stored in the windows world;

· Text Files: .ini .xml .etc

· Registry: Software Settings, Hardware Configurations, DLL registrations, File associations. Not all aspects of the registry can be managed directly since some parts are dark and badly documented. Often there are API’s available for these dark places.

· NTFS: File System configurations, ACL’s, Auditing settings, directory structures.

· Active Directory: Users/Computers, Domain Configurations, Configurations of Distributed Applications like exchange.

· Databases: MSSQL, Oracle etc

· Proprietary Databases: JET; Metabase, Shadow Copies etc., often API available.

· COM+ Configurations (Basically stored in registry, but API’s available to make it more human to configure, almost impossible to manage directly from registry)

· Global Assembly Cache aka GAC: Stores information about the public (Shared) .net libraries.

· Application Programming Interface, some configurations are stored encrypted or are hard to reverse engineer, and can only accessed via API’s.

What do we use for configuration Management

A while ago we did a PoC with Bladelogic Configuration Manager nowadays owned by BMC. Bladelogic proved to be very usefull and helps us greatly with managing the lifecycle of a Windows server.

We are able to baremetal deploy machines according to our Best Practices, with all the needed baseline software installed. From here we are able to deploy applications to our standards, capture configurations and when needed we can parameterize settings to match Development, Test, Acceptance and Production environments. When the deployment cycle has been completed, we are also able to monitor the application configuration were needed for unauthorized changes.

Besides Windows Bladelogic Configuration Manager is able to manage all major Unixes and VMware ESX/ESXi. using a native agent or via vCenter

More info: http://www.bladelogic.com