Archive

Archive for the ‘Troubleshooting’ Category

ESXi: issues with NFS datastore. Where do I put my tcpdump?

February 4th, 2011 1 comment

ESXi over NFS works just great!

But what if you have an issue with NFS and you need a network dump? 

In ESXi tipically you don’t have a local datastore where you can write files from the network dump and your datastore over NFS is not availabe!

Before running into the Data Centre and stick a USB disk or even better a SCSI disk you might want to try this. ;-)

One trick I used that worked out pretty well for me, with a little help of my a linux machine, is to send the tcpdump output to a FIFO and from a remote host (might be a VM in a different ESXi host) over SSH cat the FIFO to a local file.

How To:
On the ESXi host logon via SSH as root and create a named pipe:

root@yourESXihost# mkfifo /tmp/pipe.dmp

and from a remote linux machine launch the following:

you@yourlinuxhost > ssh root@youresxihost "cat /tmp/pipe.dmp" > capture-for-wireshark.cap

Now from a new ssh session to ESXi as root lauch

root@yourESXihost# tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp

OR even better from the remote machine:

you@yourlinuxhost > ssh root@youresxihost "tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp"
(replace the # with the proper vmk port number)

Reproduce your issue and when you finished just hit  “Cotrol+C” to stop the network dump and the cat.
Now you can open your file directly in wireshark (that’s what I use at least!)

This little trick of course can be used to troubleshoot network problems in a VM as well, dumping the traffic from a VMK# nic for the entire dvPortGroup. You just need to make sure that the the VM’s vNIC and the vmk# nic are connected to the same dvPortGroup and you must remember to allow promiscuous mode (not allowed by default)

Good Luck!

Please note: your network can be very chatty so the file can grow very fast and/or your ESXi host might not like the tcpdump so use it at your own risk and only if you really know what you are doing!

Remove Vista/W7 hardwareprofiles

March 18th, 2010 No comments

For whatever reason Microsoft removed the accessible way of removing hardware profiles in Vista and Windows 7. To help out a collegue with a corrupt profile I made a tiny Powershell script to perform this action. It’s easily done by hand in the registry, the downside is you won’t see the profile descriptions which is a tad errorprone. Also, feel free to use this but I will take no responsibility whatsoever if you use this and you break your Windows install :-)

# 20091218 – vo.o1 – PZO    – Initial hack to delete hardware profiles in Windows Vista/7
#
#————————————————————————————————————————————–
# Let’s see which profiles exist..
#————————————————————————————————————————————–
$i = 0
Write-Host “”
Write-Host “The following hardware profiles have been found on this computer:” -f white
foreach ($profile in (ls -path “HKLM:\SYSTEM\CurrentControlSet\Control\IDConfigDB\Hardware Profiles\”) ) {
Write-Host 000$i – (get-itemproperty -path “HKLM:\SYSTEM\CurrentControlSet\Control\IDConfigDB\Hardware Profiles\000$i”).FriendlyName
$i++
}
#————————————————————————————————————————————–
# Now we can ask which to remove..
#————————————————————————————————————————————–
Write-Host “”
Write-Host “You are strongly advised not to remove profile 0000 – New Hardware Profile” -f red
Write-Host “”
$input = read-host “Which profile is causing you headaches and should be removed?”
Write-Host “deleting.. “HKLM:\SYSTEM\CurrentControlSet\Hardware Profiles\$input”"
Remove-Item -Path “HKLM:\SYSTEM\CurrentControlSet\Hardware Profiles\$input”

Finally: do _not_ remove profile 0000 unless you know what you are doing. YMMV!

[BBG]

Impact of TCP offload and ‘Received Side Scaling’ on traffic handling

March 9th, 2010 No comments
 

While doing a performance test on one of our customer environments we observed the impact of TCP offload and “Receive Side Scaling” (RSS) settings on the interface card on Windows web servers in combination with traffic handling.

Setup:

1. 2x Mercury Load Runner generators hitting public URL of customer

2. Served by 3x Windows2003 SP2 servers, running IIS6

3. Load being balanced by Cisco CSS11503 to web farm.

 

The CPU performance graph of the web servers with TCP offload and RSS enabled on the internet facing (FRONT) interface:image1-with-tcp-offload-enabled

 

Similarly but a more outdated graph even more clearly showing that traffic is alternating from one web server to another:

image1-1-with-TCP-offloading-enabled

 

Most interesting right!?

What makes this traffic to alternate if the load balancer has been set up to distribute the load evenly across the farm resp each Load Runner vuser to clear its cookies and session cache after each request?

We then stumbled over this read, knowing that TCP offload to network card is a classic one , but still:
http://blogs.msdn.com/psssql/archive/2010/02/21/tcp-offloading-again.aspx

And found out the characteristic that when TCP offload and RSS were disabled, the load is more evenly spread across the web farm:

 image2-with-tcp-offload-disabled

I find this pretty cool.

Any comments?

 

Country Drink Tech-Ed 2009

November 13th, 2009 No comments
Well, we had a great party last night!
It was great seeing everyone was enjoying themselves. Of course a Dutch country drink would not be complete without some Dutch entertainment, Peter Beense gave a brilliant performance! Even some Tech-Ed speakers did attend even our Powershell dude Jeffrey Snover was spotted. Club restaurant Dante never hosted such great party before. J Needless to say Schuberg Philis was happy to endorse and sponsor this event and hope we were able to explain that work hard play hard, is our kind of game. In other words, working here is as much fun you can have with your pants on.

You can check out the photos on:
http://www.saycheese.eu/nl/events/2009/november/countrydrink

Today we closed the event with Case of the Unexplained… Windows Troubleshooting with Mark Russinovich.
A really nice session over debugging and troubleshooting crappy apps and sluggish windows systems.
Cool stuff every engineer should be able to use. We’ll digest all the tracks we’ve seen this week and post some more in the coming weeks. See you all next year!

Cheers from Berlin!

 

 

MS CLM 2007: PKCS#11 smart card self-service control error: Invalid Signature.

June 30th, 2009 No comments

I got this error (PKCS#11 smart card self-service control error: Invalid Signature.) while enrolling a certificate onto my Smartcard using Microsft CLM 2007 FP1.
Looks like Microsoft Certificate Lifecycle Manager FP1 is not yet compatible with Windows 7 RC1. No issue however with Vista SP2.
Only enrolling seems to be an issue, all other functionality just seems to work… if I find I solution to this then i will post it here… If you’ve got an solution? please leave a comment!

clm2007-error

Windows XP Mode on Windows 7 could be used as workaround as well (The smart card is accessible from XP Mode!). But the following requirements must be met for this to work;

  • Virtual XP Machine needs to be a domain member
  • Have the CLM Client tools installed
  • Have the SmartCard middleware installed.

clm-xp-mode

DFSR Debug Logging Explained

June 18th, 2009 No comments

While troubleshooting some DFSR today, I came across this very nice and detailed post from the Directory Services Team.

From: http://blogs.technet.com/askds/archive/2009/03/23/understanding-dfsr-debug-logging-part-1-logging-levels-log-format-guid-s.aspx

Ned here again. Today begins a 21-part series on using the DFSR debug logs to further your understanding of Distributed File System Replication. While there are specific troubleshooting scenarios that will be covered, the most important part of understanding any products logging is making sure you are comfortable with it before you have errors. That way you have some point of reference if things go wrong.

As you can probably guess, these posts were a long time in development. They are based on an internal DFSR whitepaper I have worked on for six months, and which went through review by a number of excellent folks here in Support, Field Engineering, and the Product Group itself. Except for the removal of all private source code references, this series is otherwise unchanged.

I’ll start with a couple posts on the logs themselves, how they are formatted, how they can be controlled, etc. Then I’ll dig into scenarios in detail, for both Windows Server 2003 R2 and Windows Server 2008. Don’t feel like you have to read and memorize everything – this series is a reference guide as well.

Understanding DFSR debug logging (Part 1: Logging Levels, Log Format, GUID’s)
Understanding DFSR debug logging (Part 2: Nested Fields, Module ID’s)
Understanding DFSR debug logging (Part 3: The Log Scenario Format, File Added to Replicated Folder on Windows Server 2008)
Understanding DFSR debug logging (Part 4: A Very Small File Added to Replicated Folder on Windows Server 2008)
Understanding DFSR debug logging (Part 5: File Modified on Windows Server 2003 R2)
Understanding DFSR debug logging (Part 6: Microsoft Office Word 97-2003 File Modified on Windows Server 2008)
Understanding DFSR debug logging (Part 7: Microsoft Office Word 2007 File Modified on Windows Server 2008)
Understanding DFSR debug logging (Part 8: File Deleted from Windows Server 2003 R2)
Understanding DFSR debug logging (Part 9: File is Renamed on Windows Server 2003 R2)
Understanding DFSR debug logging (Part 10: File Conflicted between two Windows Server 2008)
Understanding DFSR debug logging (Part 11: Directory created on Windows Server 2003 R2)
Understanding DFSR debug logging (Part 12: Domain Controller Bind and Config Polling on Windows Server 2008)
Understanding DFSR debug logging (part 13: A New Replication Group and Replicated Folder between two Windows Server 2008 members)
Understanding DFSR debug logging (Part 14: A sharing violation due to a file locked upstream between two Windows Server 2008)
Understanding DFSR debug logging (Part 15: Pre-Seeded Data Usage during Initial Sync)
Understanding DFSR debug logging (Part 16: File modification with RDC in very granular detail (uses debug severity 5))
Understanding DFSR debug logging (Part 17: Replication failing because of blocked RPC ports (uses debug severity 5))
Understanding DFSR debug logging (Part 18: LDAP queries failing due to network (uses debug severity 5))
Understanding DFSR debug logging (Part 19: File Blocked Inbound by a File Screen Filter Driver (uses debug severity 5))
Understanding DFSR debug logging (Part 20: Skipped temporary and filtered files (uses debug severity 5))
Understanding DFSR debug logging (Part 21: File replication performance from throttling (uses debug severity 5))


Dtrace for Windows? Windows Performance Toolkit

June 17th, 2009 No comments

So you have performance troubles on Windows, you probably already pulled the sysinternals from the shelve. But did you already know the Windows Performance toolkit for hardcore performance troubleshooting?

This toolkit has three tools;

xperf.exe – Captures traces, post-processes them for use on any machine, and supports command-line (action-based) trace analysis.

xperfview.exe – Visual Trace Analysis Tool – Presents trace content in the form of interactive graphs and summary tables

xbootmgr.exe – Automates on/off state transitions and captures traces during these transitions.

So what do these tools do?

Performance Analyzer is built on top of the Event Tracing for Windows (ETW) infrastructure. ETW enables Windows and applications to efficiently generate events, which can be enabled and disabled at any time without requiring system or process restarts. ETW collects requested kernel events and saves them to one or more files referred to as “trace files” or “traces.” These kernel events provide extensive details about the operation of the system. Some of the most important and useful kernel events available for capture and analysis are context switches, interrupts, deferred procedure calls, process and thread creation and destruction, disk I/Os, hard faults, processor P-State transitions, and registry operations, though there are many others.

One of the great features of ETW, supported in WPT, is the support of symbol decoding, sample profiling, and capture of call stacks on kernel events. These features provide very rich and detailed views into the system operation. WPT also supports automated perf testing. Specifically, xperf is designed for scripting from the command line and can be employed in automated performance gating infrastructures (it is the core of Windows PerfGates). xperf can also dump the trace data to an ANSI text file, which allows you to write your own trace processing tools that can look for performance problems and regressions from previous tests.

More info:

http://blogs.msdn.com/ntdebugging/archive/2008/04/03/windows-performance-toolkit-xperf.aspx
http://msdn.microsoft.com/en-us/performance/cc825801.aspx
http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/COR-T594_WH08.pptx

Download the tools here:

YouTube Preview Image