Hyper-V Network Virtualization Test

To extend further on my earlier basic testing on Hyper-V network virtualization, I have included cross-sites connection with physical network that is not on Hyper-V. This is my test setup:

Both HQ-HyperV-N1 and HQ-HyperV-N2 Hyper-V hosts are sitted on HQ site and DR-HyperV-N3 is on DR site. The necessary static routes (e.g. route add) should be configured on each host to access different IP subnets. All the VMs are pre-fixed with "Test*". TestVM01-3 VMs are hosted across IP subnets but they can reach each other on same L2 virtual subnet of 10.0.0.0/24 (denoted by blue VSID: 5001). Each VM is assigned with a pair of virtual IP/MAC that is mapped to a physical host address. Test-VM is the virtual gateway (red VSID: 5002) hosted on HQ-HyperV-N1 that routes traffic on virtual network to the physical network (vice-versa). Hereby, you may download my test powershell script and reference to the above network diagram.

Besides static routes, do take note that forwarding must be enabled on the network adapters of the Gateway (e.g. Test-GW) for routing. The forwarding can be enabled using the cmdlet "Set-NetIPInterface -InterfaceIndex [int-index] -AddressFamily IPv4 -Forwarding Enabled" for each adapter. However, the forwarding would still remain disabled upon host reboot. To make things easier, I added a feature "Remote Access" and enabled routing, so that the network adapters would always be in forwarding mode.

Mount Mailbox Database in Exchange 2010

My test mailbox servers lost all iSCSI connections abruptly. Upon restore, the mailbox database (DAG) was not mounted

According to this Technet for Exchange 2010, I should execute this cmdlet to get the MailDB01 mounted: 

Mount-Database -Identity Mailbox01\MailDB01 

But it didn't work. Instead, I had this error:

Cannot process argument transformation on parameter 'Identity'. Cannot convert value "mailbox01\maildb01" to type "Microsoft.Exchange.Configuration.Tasks.DatabaseIdParameter". Error: "'mailbox01\maildb01' is not a valid value for the identity.

Parameter name: Identity"

+ CategoryInfo : InvalidData: (:) [Mount-Database], ParameterBindin...mationException

+ FullyQualifiedErrorId : ParameterArgumentTransformationError,Mount-Database  

It must be another straight 'copy-and-paste' from Exchange 2007 documentation. Instead, the correct cmdlet for Exchange 2010 should be:
Get-MailboxDatabase -Identity MailDB01 | Mount-Database -force 

The "-force" option may be needed to ignore other previous errors. Once the database is mounted, I have to reseed the second failed copy. For further information, see "How to Reseed a Failed Mailbox Database Copy in Exchange Server 2010".

Enabling Hyper-V Network Virtualization to route across IP subnets in Provider Address Space

I've just tested out the new Microsoft SDN (a.k.a network virtualization) in Hyper-V 3.0. In summary, Hyper-V network virtualization enabled different VMs to be connected on the same L2 virtual subnet, even though the underlying Hyper-V hosts are running across various IP typologies  If you're looking for GUI to enable this new feature, you'll be in for disappointment, as SCVMM 2012 has yet to incorporate it at this point (except SCVMM SP1 Beta). Only PowerShell cmdlets can be used at the moment. To make things easier, there is a demo script available for you to download and modify. The script can be used without modification provided you've setup the Hyper-V infrastructure according to the given instructions. The demo setup should look like this:

According to above diagram, the blue VMs belong to blue tenant and the red VMs belong to the red tenant. Each respective color VM can only connect to same color VM residing on another host on the same virtual subnet of "10.0.0.0/24". Despite the overlapping IP address range, both color networks are virtually segregated by different VSIDs. Each pair of virtual IP and MAC address has to be "registered" with the underlying host IP address (i.e. Provider address) using New-NetVirtualizationLookupRecord cmdlet.

Notably, both Hyper-V hosts are connected on the same L2 subnets i.e. 192.168.4.0/24 (known as Provider Address" in the demo, which is so "unreal". In the real world, it's far more likely for providers hosts to be connected across different IP topology and subnets. Let's assume Host 2 be placed on "192.168.6.0/24". Using the same script won't suffice, you'll have to use the "New-NetVirtualizationProviderRoute" cmdlet to inform the underlying hosts how to reach the cross subnet.  After changing Host 2 provider address and adding a gateway router in-between both hosts, the new network setup should look like this:

Let's modify the original script (replace with actual Host and Gateway addresses accordingly):

1) On line 287, from:  New-NetVirtualizationProviderAddress -ProviderAddress "192.168.4.22" change to New-NetVirtualizationProviderAddress -ProviderAddress "192.168.6.22" to reflect the change in Host-2 physical IP address.

2) After line 274, add this for Host 1:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.6.0/24" -NextHop "192.168.4.1" -CimSession $Hosts[0]

3) After line 287, add this for Host 2:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.4.0/24" -NextHop "192.168.6.1" -CimSession $Hosts[1]

4) Clear the existing records by running "Remove-NetVirtualizationLookupRecord" on both hosts, re-run the script and both VMs should be able to ping each other again.

Calculation of Hyper-V Resource Utilization

Someone asked me about Hyper-V resource utilization. One can assign up-to 4 virtual processors (VP) or  vCPU per VM in Hyper-V 2008. And this VM suffered from unacceptably slow performance due to high CPU utilization. There were only a couple of VMs running per host. If he were to add more CPU cores to the parent host, would performance improve? 

The answer is most probably not unless you've already over-packed the host with many VMs.  Even if the host parent have many more CPU cores, it couldn't assign more than 1 physical CPU core to 1 VP. Although Microsoft uses the term "Logical Processor" (LP) instead of CPU core, it's an open secret that one LP represents one CPU core.

Let's take an example. We have a  Hyper-V cluster setup consisting of 3 nodes of Hyper-V host. This setup is mainly running RemoteApp (terminal servers) applications. Each node is running Windows Server 2008 R2. The hardware profile of each node consists of  4 x 6 CPU cores.

Each node is running 3 VMs (terminal servers) or total 9 VMs for the entire Hyper-V cluster. The CPU utilization of each VM is averaging 95-100% and many users have complained of slowness in performance. However, the CPU utilization of each parent host is only 50% for each host. In other words, all the VMs are only utilizing half of the 4 x 6 CPU cores per host, even though each VM is nearly saturated.

This is the “math”:

1)      Total no. of nodes in cluster = 3

2)      No. of logical processors (LP) per node = 4 x 6 CPU cores = 24

3)      Ratio of virtual CPU (vCPU) to LP (for best performance) = 1:1

4)      Max no. of vCPU per VM = 4

5)      No. of terminal server VMs per node = 3

6)      Current CPU resource utilized by all VMs per node = 4 x 3 = 12 LPs

Based on above, we calculated that our current CPU utilized by Hyper-V is just 50% per node.

Unused utilization = max no. of LP per node – utilized LP per node = 24 -  12 = 12 (or 50%). When we checked the resource monitor on each parent host, it is indeed the number. A further check with Microsoft support confirms this understanding.

Hence, my recommendation to that someone is to
1) assign more VMs per host, so that more CPU resource can be utilized; or
2) upgrade the Hyper-V hosts to Windows Server 2012 where up-to 64 vCPU can be assigned to each VM

Basic Networking of SCVMM 2012 SP1 Beta

The new Windows Server 2012 has a new feature that supports NIC teaming natively. After configuring Hyper-V NIC teaming on two hosts, I clustered them up. Everything was working well and fine. SCVMM 2012 SP1 beta is required to manage the newly minted Hyper-V cluster.

After setting up the new SCVMM server successfully, adding an existing cluster is easy. I also noticed more networking profiles have to be created before I can roll a new VM.

See the differences? It does look confusing. Fortunately, the System Center team has updated this blog to explain the relationship between various physical and virtual components: Networking in VMM 2012 SP1 – Logical Networks.

Overall SCVMM 2012 SP1 Network Architecture

From Hyper-V Host Perspective

In essence, the steps to create networking in SCVMM 2012 SP1:

1. Start from creating new Logical Networks. Each Logical Network may contain multiple Network Sites. Each site may contain multiple VLANs and IP Subnets.
2. On each Hyper-V host, associate the Virtual Switches (each virtual switch represents a physical NIC on the host) to the respective Logical Networks. 
3. Create VM Network and associate it to a Logical Network.
4. Under the hardware profile of a new VM, connect the new virtual NIC to a VM network. Select the appropriate Virtual Switch available to the parent host.

Example 1: Datacenter network contains 2 VLANs - VLAN 10 and VLAN 20 on Site A and VLAN 30 and VLAN 40 on Site B using VLAN trunking on the underlying physical switches. Create a new logical network named "Data Center Network". In it, add a network site A, then both VLAN 10 and 20. Assign the IP subnet to each VLAN. Add another network site B and add both VLAN 30 and 40 on this network site.

Example 2: Storage network contains 2 physical storage switches without VLAN trunking in both sites. Create a new logical network named "iSCSI Storage Network". In it, add both network site A and network site B. Assign VLAN tag 0 to each network site and assign the relevant IP subnet.

Logical switch and port profile, on the other hand, act as container to consistently configure identical capabilities (e.g. SR-IOV, VMQ, DHCP guards etc) for network adapters across multiple hosts. Instead of configuring individual ports on each host, you can specify the capabilities in port profiles and logical switches and apply them to the appropriate adapters.

Concept of Cisco Bridge Domain Interfaces (BDI)

Today, I came across a strange configuration on a Cisco ASR router. It's called "Bridge Domain Interfaces (BDI)". I did a search on Cisco website and the configuration looked simple. But it was short on concept explanation, which simply mentioned

"Bridge domain interface is a logical interface that allows bidirectional flow of traffic between a Layer 2 bridged network and a Layer 3 routed network traffic. Bridge domain interfaces are identified by the same index as the bridge domain. Each bridge domain represents a Layer 2 broadcast domain."

What is it used for? Why do we need it? After some thoughts and experiment, it seems to me that BDI is used to "bundle" one or more physical L2 interfaces and link it to a L3 logical interface for routing. And this L3 logical interface is the BDI. As Cisco routers won't allow you to configure IP address belonging to the same L2 subnet/domain on more than one routed interfaces, BDI is probably a workaround to overcome that limitation. It also reminds me of a routed port-channel. 

Consider the following diagram:

Both physical ports (Gi0/0/0 and Gi0/0/1) are linked to the same L2 domain (e.g. VLAN 100). 

According to Cisco, 

"An Ethernet Virtual Circuit (EVC) is an end-to-end representation of a single instance of a Layer 2 service being offered by a provider to a customer. It embodies the different parameters on which the service is being offered. In the Cisco EVC Framework, the bridge domains are made up of one or more Layer 2 interfaces known as service instances. A service instance is the instantiation of an EVC on a given port on a given router. Service instance is associated with a bridge domain based on the configuration."

I would interpret that a service instance is used to represent one L2 domain. More than 1 ports can belong to the same service instance.

Config mode:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100 # get VLAN 100 tagged traffic

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identified as interface BDI 100 in below example config

Above config would create a service instance 100 that is linked to VLAN 100 L2 domain. Standard L3 config can be performed on interface BDI 100 for routing.

interface BDI100

  vrf forwarding VPNA

  ip address 1.1.1.1 255.255.255.0

  ip ospf 1 area 0

The physical interface can even join more than 1 bridge domain (up to 4096 per router). For example, connecting to VLAN 200 (also Bridge Domain 200) as well:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identify as BDI 100 in below example config

!
 service instance 200 ethernet

    encapsulation dot1q 200

    bridge-domain 200 # identified as BDI 200 

Host Cluster Over-committed with spare memory?

I've encountered an issue on SCVMM 2012. When I attempted to place a new VM on a Hyper-V cluster, there was an error "This configuration causes the host cluster to become overcommitted". I checked all the node properties and realized that there are still more than enough available RAM on each node. Why the overcommitted problem?

I came across the Technet forum page where Mike Briggs explained the memory calculation of SCVMM. First, SCVMM would sum up a new total memory requirement by adding up all memory used by all existing VMs and the new VM requirements to be deployed. It would then calculate whether the host cluster able to withstand the new requirement if the specified number of node failures are allowed to fail. The number of nodes allowed to fail is configured in the cluster reserve.

If you are confident of the overcommitted issue, simply adjust the cluster reserve number downward and the VM placement would continue successfully. The reserve number can be found on the the General tab of the host cluster properties of SCVMM console.