Rebuilding Perfmon WMI for SCVMM

Just earlier, SCVMM (great management tool for Hyper-V) reported that one of our Hyper-V cluster nodes stopped responding. We raised a ticket with Microsoft Technet. After some days of troubleshooting with the great Chinese Microsoft engineers (with my limited Chinese vocabulary), it was discovered that the Performance Monitor (Perfmon) WMI of the affected node was corrupted and hence unable to report to the SCVMM host.


To rebuild the perfmon WMI, enter the following command using elevated command prompt at the system32 prompt:

C:\Windows\system32 > lodctr /R

Re-sync the perfmon counter with WMI by running winmgmt /resyncperf.

C:\Windows\system32 > winmgmt /resyncperf

And then restart the WMI service. The parameter 'R' for lodctr must be in capital for the rebuild. This parameter is not even documented on Microsoft Technet.

Repairing System Center Data Protection Manager

Microsoft System Center Data Protection Manager (DPM) is used to protect and backup other windows systems, which is especially useful for backing up Hyper-V virtual machines. But what happen if the DPM is corrupted or broken?

In the event of corruption of the Microsoft Windows registry, system files, or the System Center Data Protection Manager (DPM) 2010 binaries, you can repair DPM by reinstalling it. Repairing DPM involves backing up existing DPM database (using DPMBackup.exe -db cmd), uninstalling DPM, reinstalling DPM and then restoring the database. See this technet article for step-by-step.

SID duplication doesn't matter?!

I have just come across a technet blog declaring that SID duplication doesn't matter, especially in a domain environment where Domain SID instead of machine SID is used. Domain SID is re-generated whenever a computer leave and re-join a domain, which is typical for disk imaging purposes. For years, we were taught to use sysprep or newsid to regenerate new SID for every cloned image.

"I realize that the news that it’s okay to have duplicate machine SIDs comes as a surprise to many, especially since changing SIDs on imaged systems has been a fundamental principle of image deployment since Windows NT’s inception. This blog post debunks the myth with facts by first describing the machine SID, explaining how Windows uses SIDs, and then showing that - with one exception - Windows never exposes a machine SID outside its computer, proving that it’s okay to have systems with the same machine SID."

Nevertheless, the blog concluded that sysprep is still necessary for Microsoft's support:

"Note that Sysprep resets other machine-specific state that, if duplicated, can cause problems for certain applications like Windows Server Update Services (WSUS), so Microsoft’s support policy will still require cloned systems to be made unique with Sysprep"

I would take this with a pinch of salt, as I did experience strange problems in the past for having duplicated SIDs. Or rather, I would interpret the statement this way - even though SID duplication per-se may not cause problems, unpredictable outcomes may still occur, as other machine-specific states are not reset. SID duplication is an indicator of such happening.

BitLocker with TPM and Cert - Brief Introduction

BitLocker is introduced on Windows Vista and Server 2008 to guard against theft of sensitive drives and cold boot attacks. In Windows 7 and Server 2008 R2, Bitlocker further introduces some enhancements, including eliminate the need of pre-creating 1.5GB partition and "BitLocker to Go" for removable media.

You can also use BitLocker with (1) Trusted Platform Module (TPM) and (2) smart card certificate for enhanced security. TPM is a microcontroller security chip embedded on motherboard to protect sensitive key materials from unauthorized tampering. TPM is used for system drive (e.g. C:\) where Windows is installed and the certificate is for data drives, including both fixed and removable media.

On the drive that Windows is installed on, BitLocker uses the Trusted Platform Module (TPM) to detect if the computer's critical startup process has been tampered with. Additionally, a PIN or startup key can be required for users to have access to the drive's data.

BitLocker uses the TPM to lock the encryption keys that protect the data. As a result, the keys cannot be accessed until the TPM has verified the state of the computer. Encrypting the entire volume protects all of the data, including the operating system itself, the Windows registry, temporary files, and the hibernation file. Because the keys needed to decrypt data remain locked by the TPM, an attacker cannot read the data just by removing your hard disk and installing it in another computer.

During the startup process, the TPM releases the key that unlocks the encrypted partition only after comparing a hash of important operating system configuration values with a snapshot taken earlier. This verifies the integrity of the Windows startup process. The key is not released if the TPM detects that your Windows installation has been tampered with.

On fixed and removable data drives, users can use smart card certificate or password to unlock BitLocker-protected drive. An administrator that has been designated a BitLocker data recovery agent is also able to use certificate to recover access to a BitLocker-protected drive.

BitLocker and TPM recovery information can also be backup to Active Directory. This ensures that encrypted data belonging to the enterprise can always be accessed by authorized users. Backing up the TPM owner information for a computer allows administrators to locally and remotely configure the TPM security hardware on that computer. As an example, an administrator might want to reset the TPM to factory defaults when decommissioning or repurposing computers.

More detailed technical documentation and guides can be obtained from this Microsoft Technet site.

More information on BitLocker & TPM Recovery.

Dell PowerConnect VLAN Interoperability with Cisco

According to this document, Cisco catalyst switches and Dell PowerConnect switches (in fact, most other non-Cisco switches e.g. Juniper etc) are not compatible on VLAN trunking i.e. single physical link that carries multiple VLANs. This is because Dell and Cisco devices have different default mechanisms for dynamic exchange of VLAN information. Cisco Catalyst switches’ default mechanism is the VLAN trunking protocol(VTP). In contrast, Dell PowerConnect switches use the GARP VLAN registration protocol (GVRP) for dynamic exchange of VLAN configuration information. Because the Dell and Cisco switches use different protocols by default, no exchange of VLAN control traffic will take place – and thus no intra-VLAN traffic will flow between the Dell and Cisco switches. By disabling VTP and enabling GVRP on the Cisco switch, it is possible to exchange intra-VLAN data and control information in mixed Dell-Cisco environments.

I have just recieved confirmation from Cisco saying this:

Unfortunately, GVRP is not supported on Catalyst 3560 or 3750 switches. GVRP is only supported on the CatOS releases on Cat4000 and Cat6500 platfroms (on select releases - Only 6000s, 5000s and 4000 switches running CatOS software support this feature.)

If you connect a 3560 or 3750 switch to a device that supports GVRP you will see unsupported messages on the switch, telling you that the device itself is not able to process the GVRP information from the hosts/neighbours and therefore it drop it.

Nevertheless, GVRP is supported on IOS Router (cGVRP):
http://www.cisco.com/en/US/partner/docs/ios/12_2sr/12_2srb/feature/guide/srbcgvrp.html
What's the hell?! CatOS is already obsolete. And IOS routers being at L3 do not even need to pass VLAN trunking information. Does it mean Cisco lock-in?

Part 1: Using WinPE & ImageX to capture & deploy System Images

Windows Preinstallation Environment (or WinPE in short) is a lightweight operating system that you can boot up to capture system images, install drivers and system troubleshooting. Think of it like the bootable MS-DOS disk in the good old days. Unlike the 16-bit MS DOS that requires separate set of drivers, WinPE leverages on Win7 drivers. Furthermore, you can make and customise it freely using Windows Automated Installation Kits (WAIK) that is also freely available from Microsoft.

There are several online resources that teach you how to use WinPE. Nevertheless, here are the summary steps that I have compiled of using WAIK to capture & deploy system image.

0) Make a bootable WinPE CD, including ImageX toolkit.

1) Install OS, drivers, applications into a standard Dell PC

2) Run sysprep, click on "generalize" and quit.

3) Boot the PC into WinPE environment.

4) Use ImageX /capture to capture image. Save the wim image to a network folder

5) Boot new PC from WinPE.

6) Perform diskpart to create at least 2 partitions (first partition ~300MB for Win7/Win2K8R2 Bitlocker)

7) Map to the network folder that contains the WIM image.

8) Apply WIM image to new PC partition using ImageX /apply command.

Booting into WinPE

Do note that the WIM image can also be used in conjunction with Windows Deployment Services (WDS)., where it can be used as an Install Image. Think of an install image like a master image that can be installed and applied on bare-metal computers. Some of the useful online resources that outlined the detailed step-by-step include:

Walkthrough: Boot Windows PE from CD-ROM

Preparing an Image Using Sysprep and ImageX

Deploying A WIM Image With ImageX

Of these, making bootable WinPE CD is the first step. When creating a new boot configuration file called BCD by using BCDEdit, the Microsoft Technet walkthrough contains command options for BCDEdit that are no longer valid with Windows 7. In particular, replace the "-" with "/". I have listed down the workable commands outlined in the walkthrough

Bcdedit /createstore c:\winpe-amd64\mount\boot\BCD

Bcdedit /store c:\winpe-amd64\mount\boot\BCD /create {bootmgr}

Bcdedit /store c:\winpe-amd64\mount\boot\BCD /set {bootmgr} device boot

Bcdedit /store c:\winpe-amd64\mount\boot\BCD /create /d “WINPE” /application osloader

The last command returns a GUID value. Substitute with this value in the following examples.

Bcdedit /store c:\winpe-amd64\mount\boot\BCD /set <GUID> osdevice boot
Bcdedit /store c:\winpe-amd64\mount\boot\BCD /set <GUID> device boot
Bcdedit /store c:\winpe-amd64\mount\boot\BCD /set <GUID> path \windows\system32\boot\winload.exe
Bcdedit /store d:\winpe-amd64\mount\boot\BCD /set <GUID> systemroot \windows
Bcdedit /store d:\winpe-amd64\mount\boot\BCD /set <GUID> winpe yes
Bcdedit /store d:\winpe-amd64\mount\boot\BCD /set <GUID> detecthal yes
Bcdedit /store d:\winpe-amd64\mount\boot\BCD /displayorder <GUID> -addlast

SolarWinds Orion Part 2 - Monitoring Interfaces

After the new node is added in part 1, all the interfaces will be listed out for you. Typically, you would pay particular attention to interfaces that are prone to congestion and link errors e.g. WAN interfaces. Solarwinds also automatically calculate the bandwidth utilization based on the default bandwidth of the interface. For example, Solarwinds assumes 1000Mbps bandwidth for Gigabit Ethernet (GE) interface and traffic usage of 8Mbps will constitute less than 1% utilization rate. What if that interface is connected to your service provider device that provides only 10Mbps? That would become 80% high utilization instead!

To customise the actual bandwidth, click on the interface and click on "edit interface" button. Check on "Custom Bandwidth" and enter the actual transmit and recieve bandwidth.