Updating Powershell Help Files

When needing help in some Powershell cmdlets, the Get-Help cmdlet is the equivalent "man" command for *nix systems. The Powershell cmdlets are updated so frequently that it is worthwhile updating the help files as well. If the Windows host is connected directly to Internet, the help file would be updated automatically. What about for systems on closed network? There is a "Save-Help" cmdlet to download the help files and "Update-Help" cmdlet for systems not connected to Internet.

For example, run "Save-Help" from a Windows system with Internet connection to download help files for all modules to a file share.

PS > Save-Help -DestinationPath \\Fileserver\PSHelp

Run "Update-Help" on a Windows system without Internet connection.

PS > Update-Help -SourcePath \\Fileserver\PSHelp

Both cmdlets are available to Powershell 3.0. To verify the current version of Powershell of your system, run "$PSVersionTable" and look at the "PSVersion" property. For example:

Powershell: Quickly view installed Windows feature

To quickly view a list of installed Windows feature using Poweshell on Windows Server:

PS > Get-WindowsFeature | Where-Object {$_.Installed -eq $true}

To filter only feature name

PS > Get-WindowsFeature | Where-Object {$_.Installed -eq $true} | Select-Object -Property Name

Cisco Multicast Expansion Table (MET) is Full

Today, some users aren't able to join multicast groups to watch their favorite video. After pinpointing to the upstream Cisco 7600 router, I noticed the following error message in the log:

 %MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum
%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1
%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)

The router is configured PIM in sparse mode and an RP manually set.

(hostname)#show platform hardware capacity multicast
L3 Multicast Resources
  Replication mode: egress
  Replication capability: Module                              Mode
                          2                                                 Egress    
                          3                                                 Egress    
                          4                                                 Egress    
                          5                                                 Egress    
  MET table Entries: Module                             Total    Used    %Used
                     3                                                65526       28        1%
                     4                                                65526       38        1%
                     5                                                32744   32744     100%

Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.

There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.

After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingress

Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.

Hyper-V Network Virtualization Test

To extend further on my earlier basic testing on Hyper-V network virtualization, I have included cross-sites connection with physical network that is not on Hyper-V. This is my test setup:

Both HQ-HyperV-N1 and HQ-HyperV-N2 Hyper-V hosts are sitted on HQ site and DR-HyperV-N3 is on DR site. The necessary static routes (e.g. route add) should be configured on each host to access different IP subnets. All the VMs are pre-fixed with "Test*". TestVM01-3 VMs are hosted across IP subnets but they can reach each other on same L2 virtual subnet of 10.0.0.0/24 (denoted by blue VSID: 5001). Each VM is assigned with a pair of virtual IP/MAC that is mapped to a physical host address. Test-VM is the virtual gateway (red VSID: 5002) hosted on HQ-HyperV-N1 that routes traffic on virtual network to the physical network (vice-versa). Hereby, you may download my test powershell script and reference to the above network diagram.

Besides static routes, do take note that forwarding must be enabled on the network adapters of the Gateway (e.g. Test-GW) for routing. The forwarding can be enabled using the cmdlet "Set-NetIPInterface -InterfaceIndex [int-index] -AddressFamily IPv4 -Forwarding Enabled" for each adapter. However, the forwarding would still remain disabled upon host reboot. To make things easier, I added a feature "Remote Access" and enabled routing, so that the network adapters would always be in forwarding mode.

Mount Mailbox Database in Exchange 2010

My test mailbox servers lost all iSCSI connections abruptly. Upon restore, the mailbox database (DAG) was not mounted

According to this Technet for Exchange 2010, I should execute this cmdlet to get the MailDB01 mounted: 

Mount-Database -Identity Mailbox01\MailDB01 

But it didn't work. Instead, I had this error:

Cannot process argument transformation on parameter 'Identity'. Cannot convert value "mailbox01\maildb01" to type "Microsoft.Exchange.Configuration.Tasks.DatabaseIdParameter". Error: "'mailbox01\maildb01' is not a valid value for the identity.

Parameter name: Identity"

+ CategoryInfo : InvalidData: (:) [Mount-Database], ParameterBindin...mationException

+ FullyQualifiedErrorId : ParameterArgumentTransformationError,Mount-Database  

It must be another straight 'copy-and-paste' from Exchange 2007 documentation. Instead, the correct cmdlet for Exchange 2010 should be:
Get-MailboxDatabase -Identity MailDB01 | Mount-Database -force 

The "-force" option may be needed to ignore other previous errors. Once the database is mounted, I have to reseed the second failed copy. For further information, see "How to Reseed a Failed Mailbox Database Copy in Exchange Server 2010".

Enabling Hyper-V Network Virtualization to route across IP subnets in Provider Address Space

I've just tested out the new Microsoft SDN (a.k.a network virtualization) in Hyper-V 3.0. In summary, Hyper-V network virtualization enabled different VMs to be connected on the same L2 virtual subnet, even though the underlying Hyper-V hosts are running across various IP typologies  If you're looking for GUI to enable this new feature, you'll be in for disappointment, as SCVMM 2012 has yet to incorporate it at this point (except SCVMM SP1 Beta). Only PowerShell cmdlets can be used at the moment. To make things easier, there is a demo script available for you to download and modify. The script can be used without modification provided you've setup the Hyper-V infrastructure according to the given instructions. The demo setup should look like this:

According to above diagram, the blue VMs belong to blue tenant and the red VMs belong to the red tenant. Each respective color VM can only connect to same color VM residing on another host on the same virtual subnet of "10.0.0.0/24". Despite the overlapping IP address range, both color networks are virtually segregated by different VSIDs. Each pair of virtual IP and MAC address has to be "registered" with the underlying host IP address (i.e. Provider address) using New-NetVirtualizationLookupRecord cmdlet.

Notably, both Hyper-V hosts are connected on the same L2 subnets i.e. 192.168.4.0/24 (known as Provider Address" in the demo, which is so "unreal". In the real world, it's far more likely for providers hosts to be connected across different IP topology and subnets. Let's assume Host 2 be placed on "192.168.6.0/24". Using the same script won't suffice, you'll have to use the "New-NetVirtualizationProviderRoute" cmdlet to inform the underlying hosts how to reach the cross subnet.  After changing Host 2 provider address and adding a gateway router in-between both hosts, the new network setup should look like this:

Let's modify the original script (replace with actual Host and Gateway addresses accordingly):

1) On line 287, from:  New-NetVirtualizationProviderAddress -ProviderAddress "192.168.4.22" change to New-NetVirtualizationProviderAddress -ProviderAddress "192.168.6.22" to reflect the change in Host-2 physical IP address.

2) After line 274, add this for Host 1:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.6.0/24" -NextHop "192.168.4.1" -CimSession $Hosts[0]

3) After line 287, add this for Host 2:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.4.0/24" -NextHop "192.168.6.1" -CimSession $Hosts[1]

4) Clear the existing records by running "Remove-NetVirtualizationLookupRecord" on both hosts, re-run the script and both VMs should be able to ping each other again.

Calculation of Hyper-V Resource Utilization

Someone asked me about Hyper-V resource utilization. One can assign up-to 4 virtual processors (VP) or  vCPU per VM in Hyper-V 2008. And this VM suffered from unacceptably slow performance due to high CPU utilization. There were only a couple of VMs running per host. If he were to add more CPU cores to the parent host, would performance improve? 

The answer is most probably not unless you've already over-packed the host with many VMs.  Even if the host parent have many more CPU cores, it couldn't assign more than 1 physical CPU core to 1 VP. Although Microsoft uses the term "Logical Processor" (LP) instead of CPU core, it's an open secret that one LP represents one CPU core.

Let's take an example. We have a  Hyper-V cluster setup consisting of 3 nodes of Hyper-V host. This setup is mainly running RemoteApp (terminal servers) applications. Each node is running Windows Server 2008 R2. The hardware profile of each node consists of  4 x 6 CPU cores.

Each node is running 3 VMs (terminal servers) or total 9 VMs for the entire Hyper-V cluster. The CPU utilization of each VM is averaging 95-100% and many users have complained of slowness in performance. However, the CPU utilization of each parent host is only 50% for each host. In other words, all the VMs are only utilizing half of the 4 x 6 CPU cores per host, even though each VM is nearly saturated.

This is the “math”:

1)      Total no. of nodes in cluster = 3

2)      No. of logical processors (LP) per node = 4 x 6 CPU cores = 24

3)      Ratio of virtual CPU (vCPU) to LP (for best performance) = 1:1

4)      Max no. of vCPU per VM = 4

5)      No. of terminal server VMs per node = 3

6)      Current CPU resource utilized by all VMs per node = 4 x 3 = 12 LPs

Based on above, we calculated that our current CPU utilized by Hyper-V is just 50% per node.

Unused utilization = max no. of LP per node – utilized LP per node = 24 -  12 = 12 (or 50%). When we checked the resource monitor on each parent host, it is indeed the number. A further check with Microsoft support confirms this understanding.

Hence, my recommendation to that someone is to
1) assign more VMs per host, so that more CPU resource can be utilized; or
2) upgrade the Hyper-V hosts to Windows Server 2012 where up-to 64 vCPU can be assigned to each VM