First experience using AWS Virtual Private Cloud

All the while, Amazon Web Services (AWS) - the leading public cloud provider - has been advocating for the demise of On-Premise Private Cloud. Its nearest alternative offering is Virtual Private Cloud (VPC) where you could really build one anytime, anywhere. Using its free usage tier, I've built a base VPC with two EC2 instances (or VMs) as depicted below.

To learn AWS, we have to understand its terminology:

1. What's EC2 Instance? It's Virtual Machine.
2. What's VPC? Virtual network on cloud where you can create multiple IP subnets on it. EC2 instances may be hosted on a VPC.
3. What's Security Group? Think of it like a L2 firewall where you can configure the network access rules e.g. only allow HTTPS to the public Web instance from Internet etc. It is associated to one or more instances.
4. What's Subnet? The usual IP subnet that we knows of. A VPC is made up of one or more subnets. You can configure which subnet is public facing and which are not. In my example, 10.10.1.0/24 is public facing and 10.10.4.0/24 to host my internal instances.
5. What's Network ACL? Think of it like the usual network ACL applied to router interfaces. The ACL is stateless, so you've to define both inbound and outbound for a particular traffic. It can be used to complement the Security Group. For example, allow inbound TCP 443 to the subnet that hosts the above Web instance.
6. What's Elastic IP (EIP)? A public IP assigned to a public facing instance, although only private IP is assigned physically on its NIC. Think of it like an NAT address on the invisible Internet gateway.

To begin with free trial:

1. Of course, create an AWS account using your credit card. Don't worry, AWS won't charge anything to your card, as long as you stay within the free usage tier. You can enable bills monitoring if you're concern that you would exceed the free tier limit. As for me, the ultimate backstopper is to make friend with the extremely friendly AWS account managers.
2. Start with the AWS Quick Start guides, especially the RD Gateway guide.
3. Create a VPC with 2 subnets - one public facing (i.e. RD Gateway for remote admin) and another private subnet to host the internal instances.
4. Launch new instances with a wide range of Amazon Machine Image (AMI) templates to select, including various Windows Server and Linux OSes.
5. Configure the Security Group to allow inbound RDP TCP 3389 for the initial setup of RD Gateway instance.
6. After the RD Gateway is successfully setup, you can tighten network security by allowing only HTTPS traffic.

So far, the usage experience on AWS is good, as though I’m working on my own private cloud. The free SDN feature provided by AWS is also almost as agile and flexible as the VMWare NSX that I've recently experimented with. I’m also impressed by the AWS powershell supports embedded in the Windows template. Most importantly, all the AWS features are well documented. The only ‘complaint’ so far is the relative slow loading of html AWS documentation (probably not hosted/cached in Singapore?)

But can AWS really replace all on-premise private cloud networks? It definitely hold promises due to its great elasticity and flexibility. The next challenge depends on how fast its metering jump, whereas in private cloud world where metering is rarely looked at (lest even use). Much like the debate of whether it's more economical of hiring taxi daily vs owning a car, which cost can be astronomical in Singapore. 

Trying out VMWare NSX Hands-On-Lab (HOL)

Just did my first lab on VMWare NSX Hands-On-Lab on network virtualization. The task is to create a logical L2 network between 2 VMs, even though they could be separated by underlying L3 physical network and even residing on different clusters. Finally, the lab will bridge the VXLAN logical switch to VLAN 100 on production network.

Let's understand the key NSX components:

1. NSX Manager is the centralized network management component of NSX, and is installed as a virtual appliance. It provides an aggregated system view.
2. NSX controller is the central control point for all logical switches within a network and maintains information of all virtual machines, hosts, logical switches, and VXLANs. The controller is running as a VM. For redundancy, a second controller can also serve as a standby VM. The controller supports two new logical switch control plane modes, Unicast and Hybrid. These modes decouple NSX from the physical network. VXLANs no longer require the physical network to support multicast.
3. NSX Edge provides network edge security and gateway services to isolate a virtualized network and to bridge or route to physical network. You can install NSX edge either as a logical router or a services gateway.
4. NSX vSwitch replaces the default Virtual Distributed Switch (VDS) on the kernel mode hypervisor on each host.

I won't go through the detailed step-by-step. Rather, I would highlight the high level steps to serve better understanding on the deployment scenario.

Step 0: Preparing the network and clusters. There are 3 clusters of hosts. Compute A and B are 2 clusters are meant for hosting VMs. NSX vSwitch resides on all hypervisors. Other NSX components mentioned above reside on the "Management and Edge Cluster".

Step 1: Enable VXLAN Tunnel End Points (VTEPs) and VXLAN using vSphere client.

Step 2: Create a VXLAN Transport Zone spanning the 3 clusters

Step 3: Create a logical switch and attach it to an NSX Edge. The Edge gateway has an interface of 192.168.100.1 connecting to the transport zone. Note the new L2 logical network (172.16.40.0/24) created in green.

Step 4: Add two Web VMs and their vNICs to the new logical network as shown below. Both static IP and DHCP should work fine on the VMs. Test connectivity between both VMs.

Step 5: Bridging the logical switch to the physical network via NSX Logical Router. In this case, the VXLAN is bridged to VLAN 100 on the production network.

Hyper-V Network Virtualization Test

To extend further on my earlier basic testing on Hyper-V network virtualization, I have included cross-sites connection with physical network that is not on Hyper-V. This is my test setup:

Both HQ-HyperV-N1 and HQ-HyperV-N2 Hyper-V hosts are sitted on HQ site and DR-HyperV-N3 is on DR site. The necessary static routes (e.g. route add) should be configured on each host to access different IP subnets. All the VMs are pre-fixed with "Test*". TestVM01-3 VMs are hosted across IP subnets but they can reach each other on same L2 virtual subnet of 10.0.0.0/24 (denoted by blue VSID: 5001). Each VM is assigned with a pair of virtual IP/MAC that is mapped to a physical host address. Test-VM is the virtual gateway (red VSID: 5002) hosted on HQ-HyperV-N1 that routes traffic on virtual network to the physical network (vice-versa). Hereby, you may download my test powershell script and reference to the above network diagram.

Besides static routes, do take note that forwarding must be enabled on the network adapters of the Gateway (e.g. Test-GW) for routing. The forwarding can be enabled using the cmdlet "Set-NetIPInterface -InterfaceIndex [int-index] -AddressFamily IPv4 -Forwarding Enabled" for each adapter. However, the forwarding would still remain disabled upon host reboot. To make things easier, I added a feature "Remote Access" and enabled routing, so that the network adapters would always be in forwarding mode.

Enabling Hyper-V Network Virtualization to route across IP subnets in Provider Address Space

I've just tested out the new Microsoft SDN (a.k.a network virtualization) in Hyper-V 3.0. In summary, Hyper-V network virtualization enabled different VMs to be connected on the same L2 virtual subnet, even though the underlying Hyper-V hosts are running across various IP typologies  If you're looking for GUI to enable this new feature, you'll be in for disappointment, as SCVMM 2012 has yet to incorporate it at this point (except SCVMM SP1 Beta). Only PowerShell cmdlets can be used at the moment. To make things easier, there is a demo script available for you to download and modify. The script can be used without modification provided you've setup the Hyper-V infrastructure according to the given instructions. The demo setup should look like this:

According to above diagram, the blue VMs belong to blue tenant and the red VMs belong to the red tenant. Each respective color VM can only connect to same color VM residing on another host on the same virtual subnet of "10.0.0.0/24". Despite the overlapping IP address range, both color networks are virtually segregated by different VSIDs. Each pair of virtual IP and MAC address has to be "registered" with the underlying host IP address (i.e. Provider address) using New-NetVirtualizationLookupRecord cmdlet.

Notably, both Hyper-V hosts are connected on the same L2 subnets i.e. 192.168.4.0/24 (known as Provider Address" in the demo, which is so "unreal". In the real world, it's far more likely for providers hosts to be connected across different IP topology and subnets. Let's assume Host 2 be placed on "192.168.6.0/24". Using the same script won't suffice, you'll have to use the "New-NetVirtualizationProviderRoute" cmdlet to inform the underlying hosts how to reach the cross subnet.  After changing Host 2 provider address and adding a gateway router in-between both hosts, the new network setup should look like this:

Let's modify the original script (replace with actual Host and Gateway addresses accordingly):

1) On line 287, from:  New-NetVirtualizationProviderAddress -ProviderAddress "192.168.4.22" change to New-NetVirtualizationProviderAddress -ProviderAddress "192.168.6.22" to reflect the change in Host-2 physical IP address.

2) After line 274, add this for Host 1:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.6.0/24" -NextHop "192.168.4.1" -CimSession $Hosts[0]

3) After line 287, add this for Host 2:

New-NetVirtualizationProviderRoute -InterfaceIndex $iface.InterfaceIndex -DestinationPrefix "192.168.4.0/24" -NextHop "192.168.6.1" -CimSession $Hosts[1]

4) Clear the existing records by running "Remove-NetVirtualizationLookupRecord" on both hosts, re-run the script and both VMs should be able to ping each other again.

Concept of Cisco Bridge Domain Interfaces (BDI)

Today, I came across a strange configuration on a Cisco ASR router. It's called "Bridge Domain Interfaces (BDI)". I did a search on Cisco website and the configuration looked simple. But it was short on concept explanation, which simply mentioned

"Bridge domain interface is a logical interface that allows bidirectional flow of traffic between a Layer 2 bridged network and a Layer 3 routed network traffic. Bridge domain interfaces are identified by the same index as the bridge domain. Each bridge domain represents a Layer 2 broadcast domain."

What is it used for? Why do we need it? After some thoughts and experiment, it seems to me that BDI is used to "bundle" one or more physical L2 interfaces and link it to a L3 logical interface for routing. And this L3 logical interface is the BDI. As Cisco routers won't allow you to configure IP address belonging to the same L2 subnet/domain on more than one routed interfaces, BDI is probably a workaround to overcome that limitation. It also reminds me of a routed port-channel. 

Consider the following diagram:

Both physical ports (Gi0/0/0 and Gi0/0/1) are linked to the same L2 domain (e.g. VLAN 100). 

According to Cisco, 

"An Ethernet Virtual Circuit (EVC) is an end-to-end representation of a single instance of a Layer 2 service being offered by a provider to a customer. It embodies the different parameters on which the service is being offered. In the Cisco EVC Framework, the bridge domains are made up of one or more Layer 2 interfaces known as service instances. A service instance is the instantiation of an EVC on a given port on a given router. Service instance is associated with a bridge domain based on the configuration."

I would interpret that a service instance is used to represent one L2 domain. More than 1 ports can belong to the same service instance.

Config mode:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100 # get VLAN 100 tagged traffic

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identified as interface BDI 100 in below example config

Above config would create a service instance 100 that is linked to VLAN 100 L2 domain. Standard L3 config can be performed on interface BDI 100 for routing.

interface BDI100

  vrf forwarding VPNA

  ip address 1.1.1.1 255.255.255.0

  ip ospf 1 area 0

The physical interface can even join more than 1 bridge domain (up to 4096 per router). For example, connecting to VLAN 200 (also Bridge Domain 200) as well:

interface range Gi0/0/0-1

  service instance 100 ethernet

    encapsulation dot1q 100

    rewrite ingress tag pop 1 symmetric #pop out all ingress VLAN 100 tags from switch

    bridge-domain 100 # identify as BDI 100 in below example config

!
 service instance 200 ethernet

    encapsulation dot1q 200

    bridge-domain 200 # identified as BDI 200 

Hyper-V Network Virtualization

In one of my earlier posts, I talked about software-based network virtualizaton called "Nicira NVP". The key feature is about multi-tenancy Data Center Interconnect (DCI) by creating multiple layer 2 virtual networks (or pseudo-wire) across an IP network. Layer 2 networks are essential for many Data Center applications, especially for the "free" movement of Virtual Machines (VMs) across sites and IP topology. 

In the new Windows Server 2012, Hyper-V offers similar network virtualization capability using NVGRE, which is another standard L2-over-L3 tunnel. In short, Hyper-V in WS8 includes a "Nicira" software component for network virtualization that allows same virtual subnet addressing across sites and IP topology. For full long-winded story and presentation, please visit this TechEd 2012 site. 

Here, I would just extract a single slide that tells all:

As for joining the network virtualized environment to the non-network virtualized environment, Hyper-V Network Virtualization gateways are required to bridge the two environments. See "Hyper-V Network Virtualization Gateway Architectural Guide".

Gateways can come in different form factors. They can be built upon Windows Server 2012, incorporated into a Top of Rack (TOR) switch, put into an existing network appliance, or can be a stand-alone network appliance. F5 has announced one such network appliance (F5 To Deliver Microsoft Network Virtualization Gateway).

If you are looking for Technet reference,  click on "Network Virtualization technical details".

Nicira Network Virtualization Platform (NVP)

Yesterday, I had a short technical discussion with a Singapore-based Nicira staff member. They have a niche solution that is doing exactly what the Cisco Nexus and L2 MPLS (i.e. Ethernet over MPLS or Any Transport over MPLS) are trying to achieve: multi-tenancy Data Center Interconnect (DCI) by creating multiple layer 2 virtual networks (or pseudo-wire) across an IP network. Layer 2 networks are essential for many Data Center applications, especially Cloud Virtualization. Imagine performing server clustering, VMWare vMotion, Hyper-V Live Migration on multiple physical sites, which otherwise break by IP subnets. It should also be seriously considered as part of IT Disaster Recovery plans. As for multi-tenancy, you may have multiple network customers or tenants sharing the same underlying physical infrastructure. Each tenant should only see its respective overlay network without visibility into other virtual network - a similar concept to host virtualization and cloud computing.

How does Nicira NVP work? From what I understand in high level perspectives, a data path STT (Stateless Transport Tunneling) tunnel is established between 2 or more Open vSwitches (OVS) across an IP network. This MAC-over-IP tunnel is used to encapsulate all MAC layer traffic and transport them over an IP network, which effectively connect 2 servers (whether virtual or physical) on different sites as if they were on the same subnet or VLAN. 

As of now, the OVS can be integrated into ESX, KVM and XenServer hypervisors. There is also near-future plan for Hyper-V support (not sure if the plan would be cancelled, as Nicira is now acquired by its arch-rival VMWare). Alternatively, an ISO-based image can also be run off as a virtual or physical server as an OVS gateway connecting legacy systems to the Nicira network virtualization platform (NVP).

As for OVS management and control, the NVP Controller Cluster (housed on server clustering) is used to centrally managing and controlling all OVS along the control paths. I was told that even if there were a disconnection on the control paths, the OVS would continue to operate (even though not modifiable at this stage).

From what I observed, the new Cisco Overlay Transport Virtualization (OTV) is probably Nicira's arch-rival at this point. True enough that traditional networking MPLS and L2 pseudo-wires can be employed to perform the same tricks, they are either limited by performance, lack of MPLS aware devices or simply staff knowledge in the IP networks. Both Cisco OTV and Nicira NVP, on the other hand, can be easily established across any traditional IP-based networks.

And prices also do matter. I was told that one would need at least USD150K for a small POC setup to "try out" Nicira NVP. For the same price tag, I could also have purchased at least half-dozen of Cisco ASR1000 hardware (OTV is now supported on ASR platform from version XE3.5S onward). A princely sum that is pretty hard-to-justify for a software purchase to the management I suppose.