Credential Roaming

Credential Roaming (CR) is especially useful in situations whereby the following 2 conditions exist: (1) user certificate auto-enrollment is enabled; (2) many user tends to logon more than 1 domain machine. You can imagine how many certificates for the same group of users are issued on your CA server if credential roaming were not enabled. No problem for domain logon, as UPN remains the same across multiple user certs. But for applications, such as S/MIME, encryption/decryption using the same cert key becomes a must for roaming users. Do note that only X.509 certificates and private keys are supported and they are stored on the Active Directory certificate store (so beware of potential NTDS bloat but that's another issue).

CR is a new alternative to Roaming User Profile (RUP) for storing user certificates and keys. According to Microsoft, CR is also supposed to be a more scalable and secure alternative than RUP. Whenever a user logon and logoff, the local user cert store would always sync with the AD cert store. Hence, the same user would always get the same cert and key regardless of what domain machines he log-on to.

Depending on your current AD level, you may or may not need to update the AD schema. For Windows 2003, refer to this guide. For Windows 2008, you can refer to this simpler guide instead.  

SDM Mismatch on Cisco Catalyst 3750 switches

Cisco stackable switch - Catalyst 3750 - allows you to stack different models together. For example, you can stack a 12-port SFP fiber switch (Cisco 3750G-12SD) with a 48-port 1000BaseT (Cisco 3750G-48T) switch. However, if you configure the fiber switch as master, you would see the SDM Mismatch error when you perform a "show switch" command. This is described in this Cisco article.

An extract:

The Catalyst 3750-12S switch supports desktop and aggregator Switch Database Management (SDM)templates. All other Catalyst 3750 switches support only the desktop SDM templates. All stack members use the SDM template configured on the stack master. If the stack master is using an aggregator template, only Catalyst 3750-12S switches can be stack members. All other switches attempting to join this switch stack enter SDM-mismatch mode. These switches can join the stack only when the stack master is running a desktop SDM template. We recommend that your stack master use an aggregator template only if you plan to create a switch stack of Catalyst 3750-12S switches. If you plan to have a switch stack with different Catalyst 3750 switch models, configure the stack master to use one of the desktop templates.

To use the desktop template, perform the following commands on the stack master:

1. For L2 configuration: sdm prefer vlan desktop
2. For L3 configuration (enabled with ip routing): sdm prefer routing desktop
3. Do a "show sdm prefer" to verify and reload.

Server NIC Teaming with Cisco Nexus

In my earlier post, a Cisco Nexus 5000 with multiple Nexus 2000 extenders can be managed as one network switch system, which simplify the management of data center network. In another post, network high availability for Windows Servers can be achieved by using NIC teaming. Linking both together, you could achieve total high availability with streamlined management for the entire data center network. Consider the setup below, the server is dual-homed to 2 different switch systems:

Using NIC Teaming with Link Aggregation mode (such as Intel ANS or Broadcom BACS), a virtual network adapter could be created by joining up the physical dual server NICs and is assigned with a single IP address. Not only the virtual adapter is bundled with double bandwidth (e.g. 2 x 1Gbps), network redundancy can also be achieved as the server is connected to 2 different switch systems. Even if any single switch component were to fail, the server would still remain connected to the corporate network.

On the other side of Cisco networking, Link Aggregation is known as EtherChannel. Traditionally, multiple switch ports on a EtherChannel can only be formed on a single switch system. It could also mean single point of failure for servers that are NIC-teamed in this manner. With Cisco Nexus, EtherChannel can be created across different switch systems on a same Virtual Port Channel (vPC) domain. A quick Cisco Nexus vPC configuration guide can be found here, which is relatively easy to follow. (Do pay particular attention to the example of figure 4, which is the closest to this setup.)

In summary, these are the overall steps:

1. Enable the vPC and LACP features.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link across the out-of-band management interfaces.
4. Create the vPC peer link across both Nexus 5000 switch systems through the EtherChannel link.
5. Create new EtherChannel for the server ports and assign it with the same vPC number on both switch systems.

Step 1 to 4 is usually performed only once during the initial vPC setup. For subsequent server setup, just repeat step 5. The configuration example for step 5 is as follows:

N5k(config-if)# int ethernet 100/1/1 (switch port of fex extender 100 on N2K)
N5k(config-if)# channel-group 10
N5k(config-if)# int po10
N5k(config-if)# vpc 10 (ensure this number must be the same on the other switch system)
N5k(config-if)# switchport access vlan 101

Commands to verify vPC configuration

Note on BACS Link Aggregation

If you were using Broadcom BACS, choose "Generic Trunking (FEC/GEC)/802.3ad-Draft Static" mode instead of "Link Aggregation". The former works well with Cisco LACP while the latter may give unpredictable results.

Cert Enrollment Failed after CA Cert Renewal with Event ID 93

After I renewed my CA cert, all cert enrollment failed with error messages saying that the user does not have the permission to read the cert template. It is not true, as I have assigned sufficient security permission with Read and Enroll on the issued template. A further look into the Event Viewer shows this:

Event 93, Certificate Authority

The certificate (#%1) of certification authority %2 does not exist in the certificate store at CN=NTAuthCertificates,CN=Public Key Services,CN=Services in the Active Directory's configuration container. The directory replication may not be completed.

Performing the troubleshooting tasks on this Technet post did not help. The solution is to publish the new CRT manually to the NTAuthCA store.

1. Login to CA using Domain Admin account.
2. Go to the %systemroot%\System32\CertSrv\CertEnroll directory.
3. Look for the latest *.crt file e.g. foo(2).crt
4. Run "certutil -dspublish foo(2).crt NTAuthCA"
5. Restart the CA service

Go to the event viewer again, you would see an informational event 26 saying that the CA has started and without the error event 93. The right cert templates appears when you perform manual enrollment.

Migration from FRS to DFRS with RODC

Traditionally, File Replication Service (FRS) is used for replicating the contents of the SYSVOL share between Windows domain controllers (a.ka. AD replication). A new AD DS feature in Windows Server 2008 can use the DFS Replication (DFSR) service for replicating the contents of the SYSVOL share. DFSR offers several advantages over FRS, including

1. Efficient, scalable and reliable file replication protocol which has been tested extensively to ensure data consistency in multi-master replication scenarios.
2. Differential replication of changes to files using the Remote Differential Compression (RDC) algorithm, which enhances efficiency in branch office scenarios.
3. Flexible scheduling and bandwidth throttling mechanisms.
4. Self-heals from USN journal wraps and database corruptions – end user intervention and monitoring requirement is minimal.
5. Provides built in health monitoring tools for ease of monitoring deployments.
6. Improved support for Read Only Domain Controllers.

Click here to find detailed step-by-step migration guide, which comprises a 5-part series. A quicker step-by-step can be found here. Generally, migration is pretty easy and straightforward. However, if you have promoted a Read-Only Domain Controller (RODC) before the migration is complete, you will see the following error:

The RODC will never move to the next step beyond 'Start'. To resolve it, perform the following command on the PDC emulator:

• dfsrmig /DeleteRoNtfrsMember: Since Read Only Domain Controllers cannot delete the FRS related global settings from their own Active Directory, this operation needs to be performed by the Primary Domain Controller in its Active Directory. When these changes eventually replicate in to the Read Only Domain Controller (after applicable AD Replication latencies), the DFS Replication service on the RODC is able to proceed with the process of migration.

When you run dfsrmig /getGlobalState again, the RODC entry will disappear. Issue repadmin /syncall on PDC emulator to sync with all domain controllers (including RODCs).

Multicast for MPLS VPN Extranet

In my earlier blog post, I've mentioned briefly about Multicast VPN across MPLS VPN. In this setup, the VRF must share the same default Multicast Distribution Tree (MDT), 239.1.1.1 in this example, for the multicast VPN to form dynamically. Consider the following setup:

The source server (on PE1) is streaming on multicast mode. The VRF-Green, on the other side of PE2, would be able to join to the same multicast distribution tree (239.1.1.1) and recieve the multicast stream. For this Intranet Multicast VPN to work, the VRF-Green on both PE routers should share the basic common configuration as follows:

Router PE1 and PE2
  ip vrf VRF-Green
  mdt default 239.1.1.1
  route-target export 65001:100
  route-target import 65001:100
!
ip multicast-routing
ip multicast-routing vrf VRF-Green
ip pim rp-address 1.1.1.1
ip pim vrf VRF-Green rp-address 10.1.1.1
!
router bgp 55
  address-family ipv4 mdt
    neighbor x.x.x.x activate
    neighbor x.x.x.x send-community extended
!
  address-family vpnv4
     neighbor x.x.x.x activate
     neighbor x.x.x.x send-community extended
!

However, the VRF-Blue (on PE2) won't be able to join the same MDT, as Cisco routers do not support multiple duplicate default MDT across VRFs. However, there is a workaround for this Extranet setup. Do note that data flow in MPLS VPN happens in label-exchange. Create another VRF-Green on the source PE1 and import the multicast traffic over to VRF-Blue on PE2. There are 2 options of importing, either import it on the source PE or on the reciever PE. I observed that the latter reciever PE option is more reliable if there are more than 1 reciever PE routers.

Configuring the Receiver MVRF on the Source PE (Option 1)
On PE1, add "route-target import 65001:100" on VRF-Blue, enable multicast routing "ip multicast-routing vrf VRF-Blue", and specify the same Rendezvous Point (RP) as VRF-Green for VRF-Blue using "ip pim vrf VRF-Blue rp-address 10.1.1.1".

For this setup, assign a new default MDT group for VRF-Blue on both PE routers (239.2.2.2 in this example). Import the label on VRF-Green over (route-target import 65001:100) as shown in above diagram. In this way, two multicast distribution trees (MDTs) can be created from the same source server. When you perform a "show ip mroute vrf VRF-Green" on PE1, you'll see Extranet receivers in VRF-Blue.

Configuring the Source MVRF on the Receiver PE (Option 2)
Another option is to do the import on the reciever PE (PE 2 in this example). As mentioned earlier, I observed that the stream is slightly more stable if there is more than 1 reciever PE. You'll need to do the import on every reciever PE.

Troubleshooting
Ensure RPF and multicast routing is sound for GRT level on all PE and P routers, as well as VRF level for all PE. In this example (some outputs truncated):
GRT level
#sh ip mroute
(10.10.10.1, 239.1.1.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Port-channel1, RPF nbr 10.10.0.1
  Outgoing interface list:
    Port-channel2, Forward/Sparse, 00:54:29/00:03:15

VRF level
#sh ip mroute vrf VRF-Green
(192.168.1.1, 239.23.25.1), 00:54:29/00:02:12, flags: TA
  Incoming interface: Tunnel1, RPF nbr 192.168.0.1
  Outgoing interface list:
    VLAN123, Forward/Sparse, 00:54:29/00:03:15

For further details, refer to the Cisco Multicast VPN Extranet support.

Multipath with Redundant Routers
If you have redundant paths with some load splitting or sharing, you may have to consider enabling multipath options using following steps on every router:

1. enable
2. configure terminal
3. ip multicast multipath s-g-hash next-hop-based
4. ip multicast vrf vrf-Name multipath s-g-hash next-hop-based
5. end
6. show ip rpf source-address group-address
7. show ip route ip-address

Reference: Load Splitting IP Multicast Traffic over ECMP