Misconfigured Interfaces on NLB cluster of Hyper-V VMs

If you ever attempt to form a Network Load Balancing (NLB) cluster with 2 or more Hyper-V VMs, you would be likely to see errors saying that the interfaces are misconfigured and both cluster and dedicated IP addressess are unable to be added to TCP/IP.

To workaround this issue, shut down the VMs. Open up Hyper-V manager and enable MAC spoofing on the interfaces that you want to load-balance as follows:

Start the VMs. Remove the old cluster and form a new one.

Windows Date for Backup Batch Jobs

When you perform "echo %date%" on your Windows command prompt, you would see a date format unsuitable for backup batch job. Try this instead:

echo %date:~-4,4%%date:~4,2%%date:~7,2%

You may want to manipulate the numeric positions (negative sign "-" means reverse position starting from right-end) according to your needs.

Credential Roaming

Credential Roaming (CR) is especially useful in situations whereby the following 2 conditions exist: (1) user certificate auto-enrollment is enabled; (2) many user tends to logon more than 1 domain machine. You can imagine how many certificates for the same group of users are issued on your CA server if credential roaming were not enabled. No problem for domain logon, as UPN remains the same across multiple user certs. But for applications, such as S/MIME, encryption/decryption using the same cert key becomes a must for roaming users. Do note that only X.509 certificates and private keys are supported and they are stored on the Active Directory certificate store (so beware of potential NTDS bloat but that's another issue).

CR is a new alternative to Roaming User Profile (RUP) for storing user certificates and keys. According to Microsoft, CR is also supposed to be a more scalable and secure alternative than RUP. Whenever a user logon and logoff, the local user cert store would always sync with the AD cert store. Hence, the same user would always get the same cert and key regardless of what domain machines he log-on to.

Depending on your current AD level, you may or may not need to update the AD schema. For Windows 2003, refer to this guide. For Windows 2008, you can refer to this simpler guide instead.  

SDM Mismatch on Cisco Catalyst 3750 switches

Cisco stackable switch - Catalyst 3750 - allows you to stack different models together. For example, you can stack a 12-port SFP fiber switch (Cisco 3750G-12SD) with a 48-port 1000BaseT (Cisco 3750G-48T) switch. However, if you configure the fiber switch as master, you would see the SDM Mismatch error when you perform a "show switch" command. This is described in this Cisco article.

An extract:

The Catalyst 3750-12S switch supports desktop and aggregator Switch Database Management (SDM)templates. All other Catalyst 3750 switches support only the desktop SDM templates. All stack members use the SDM template configured on the stack master. If the stack master is using an aggregator template, only Catalyst 3750-12S switches can be stack members. All other switches attempting to join this switch stack enter SDM-mismatch mode. These switches can join the stack only when the stack master is running a desktop SDM template. We recommend that your stack master use an aggregator template only if you plan to create a switch stack of Catalyst 3750-12S switches. If you plan to have a switch stack with different Catalyst 3750 switch models, configure the stack master to use one of the desktop templates.

To use the desktop template, perform the following commands on the stack master:

1. For L2 configuration: sdm prefer vlan desktop
2. For L3 configuration (enabled with ip routing): sdm prefer routing desktop
3. Do a "show sdm prefer" to verify and reload.

Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.

Cert Enrollment Failed after CA Cert Renewal with Event ID 93

After I renewed my CA cert, all cert enrollment failed with error messages saying that the user does not have the permission to read the cert template. It is not true, as I have assigned sufficient security permission with Read and Enroll on the issued template. A further look into the Event Viewer shows this:

Event 93, Certificate Authority

The certificate (#%1) of certification authority %2 does not exist in the certificate store at CN=NTAuthCertificates,CN=Public Key Services,CN=Services in the Active Directory's configuration container. The directory replication may not be completed.

Performing the troubleshooting tasks on this Technet post did not help. The solution is to publish the new CRT manually to the NTAuthCA store.

1. Login to CA using Domain Admin account.
2. Go to the %systemroot%\System32\CertSrv\CertEnroll directory.
3. Look for the latest *.crt file e.g. foo(2).crt
4. Run "certutil -dspublish foo(2).crt NTAuthCA"
5. Restart the CA service

Go to the event viewer again, you would see an informational event 26 saying that the CA has started and without the error event 93. The right cert templates appears when you perform manual enrollment.

Migration from FRS to DFRS with RODC

Traditionally, File Replication Service (FRS) is used for replicating the contents of the SYSVOL share between Windows domain controllers (a.ka. AD replication). A new AD DS feature in Windows Server 2008 can use the DFS Replication (DFSR) service for replicating the contents of the SYSVOL share. DFSR offers several advantages over FRS, including

1. Efficient, scalable and reliable file replication protocol which has been tested extensively to ensure data consistency in multi-master replication scenarios.
2. Differential replication of changes to files using the Remote Differential Compression (RDC) algorithm, which enhances efficiency in branch office scenarios.
3. Flexible scheduling and bandwidth throttling mechanisms.
4. Self-heals from USN journal wraps and database corruptions – end user intervention and monitoring requirement is minimal.
5. Provides built in health monitoring tools for ease of monitoring deployments.
6. Improved support for Read Only Domain Controllers.

Click here to find detailed step-by-step migration guide, which comprises a 5-part series. A quicker step-by-step can be found here. Generally, migration is pretty easy and straightforward. However, if you have promoted a Read-Only Domain Controller (RODC) before the migration is complete, you will see the following error:

The RODC will never move to the next step beyond 'Start'. To resolve it, perform the following command on the PDC emulator:

• dfsrmig /DeleteRoNtfrsMember: Since Read Only Domain Controllers cannot delete the FRS related global settings from their own Active Directory, this operation needs to be performed by the Primary Domain Controller in its Active Directory. When these changes eventually replicate in to the Read Only Domain Controller (after applicable AD Replication latencies), the DFS Replication service on the RODC is able to proceed with the process of migration.

When you run dfsrmig /getGlobalState again, the RODC entry will disappear. Issue repadmin /syncall on PDC emulator to sync with all domain controllers (including RODCs).