Installation of SCOM 2007 R2

System Center Operations Manager (SCOM) 2007 R2 is the latest monitoring platform from Microsoft that manage hetergenous systems and applications, including both Microsoft and non-Microsoft. SCOM relies on Management Pack (MP) created by its developers and partners to monitor specific systems and applications.

The components of SCOM 2007 R2 should be installed in the following orders:

1. Operations Database Server - MS SQL server for data storage
2. Root Management Server (RMS) and console - First monitoring server
3. Management Servers - Subsequent monitoring servers
4. Reporting Server and Data Warehouse Server - Generate reports
5. ACS Database Server and ACS Collection Server - Audit Collection Services for storing security events and logs
6. Gateway Server - Collect data from multiple agents residing outside of trusted internal domain.
7. Agentless Exception Monitoring - Collecting crash information from computers without SCOM agents
8. Operations Manager Agents - Agents residing on managed systems. Collect system information and send to management server
9. ACS Forwarders

For a quick-start, this technet blog depicted step-by-step SCOM installation guide with pictures. Nevertheless, MS SQL Server 2008 must first be installed (MS SQL 2008 step-by-step).

Updating of SCVMM Template in Offline mode

The new Virtual Machine Servicing Tool (VMST) 3.0 helps to service and patch Hyper-V VMs in offline mode. The new v3.0 introduces the following new features:

1. Offline virtual machines in a SCVMM library.
2. Stopped and saved state virtual machines on a host.
3. Updating of Virtual machine templates.
4. Offline virtual hard disks in a SCVMM library by injecting update packages.

The new feature that I’m waiting for is the ‘offline’ patching of VM templates. In SCVMM, template is an efficient way to deploy pre-configured VMs quickly over and over again. However, the newly created VMs out of a particular template will be ‘out-of-patch’ soon. Imagine you can’t enjoy the new features offered in Windows 2008 R2 SP1 immediately, as the VM template was created quite sometime ago.

For downloading of this tool, click here. I won’t go into details, as a “Getting Started” guide is included in the toolkit. Hence, I would just summarize the procedure of servicing VM template (the feature that I’m most interested on):

1. Prepare the infrastructure for servicing, including AD, DHCP, SCVMM, and updating services (WSUS/SCCM). I believe most of us should already have this infrastructure in place if you’re using Hyper-V and SCVMM seriously.
2. The Virtual Machine Servicing Tool must be installed on the same server as the VMM Administrator Console (the Administrator Console provides Windows PowerShell support). The VMM server and library components can reside on the same server or on one or more additional servers.
3. From the target template that you want to update offline, use SCVMM to create a Gold VM out of it. VMST will deploy the Gold VM on a maintenance Hyper-V host with WSUS/SCVMM patches and replace it with an updated VM template eventually. The Gold VM must be stored in the same VMM library.
4. Perform initial setup on VMST to reference the VMM server, Maintenance Host, update server (WSUS and/or SCCM server).
5. Create a Template Group. Associate the Gold VM to this template.
6. Create scheduled servicing jobs. Start the job.

During the servicing job, the following tasks will occur:

1. Mounts the gold virtual machine associated with the virtual machine template on the maintenance host.
2. Starts the gold virtual machine on the maintenance host, and then invokes either Configuration Manager or WSUS to update it.
3. Clones the updated gold virtual machine, creates a template from the cloned virtual machine, and then uses Sysprep to generalize the cloned gold virtual machine to remove hardware ID information.
4. Shuts down the updated gold virtual machine, and returns it to the offline VMM Library.
5. Associates the .vhd file of the cloned virtual machine template with the original template, and then deletes the cloned virtual machine template.

Resetting Computer Account

A domain computer account synchronizes with the Domain Controller (DC) on a regular basis. This means that the computer checks with the DC or the DC checks for the computer on the network at a set interval. If for some reason synchronization does not take place, then the computer account can become invalid due to failed authentication. Group policy may also fail to take effect.

Each domain computer maintains a machine account password history containing the current and previous passwords used for the account. When the computer attempt to authenticate with DC and a change to the current password is not yet received, Windows then relies on the previous password. If this authentication fails (due to the failed sync of password), both computers may not communicate. Hence, you have to reset the computer password. You can't set the password directly but you can perform a computer account reset on the "Active Directory User and Computer" console or "netdom reset" on a DC.

After you have reset the computer account, you won't be able to login to the affected computer using domain-based accounts. You have to re-join the computer to domain, so that the AD re-sync can take place. Login to the affected computer on local administrator account. At the elevated command prompt,

1. Force leave the domain: netdom remove %computername% /domain:{domain-name} /force
2. Re-join the domain: netdom join %computername% /domain:{domain-name} /UserD:{domain user} /PasswordD:{domain password}
3. Reboot the computer

Part 3: Network installation using WDS

On part 2, we have captured the new install image for new computer deployment. We're ready to perform PXE boot for new computer deployment. For better security, we should configure the PXE response to known computer only. Otherwise, any machines can be installed and joined to your domain without you knowing. If unused, the WDS server should be shut off.

To identify "known" PXE clients, you would have to pre-stage or pre-create the new computer accounts using the RSAT AD tools on the WDS computer. Install the AD tools using "Add Feature" of the Server Manager. If you pre-create from a Domain Controller, the "Next" button and "Managed Computer" page would be missing and you can't enter the GUID or MAC addresses (prefixed with 20 zeroes) of the new machine.

There is this nice blog post that outlined the detailed step-by-step.

Part 2: Forefront TMG URL Filtering

Following up on part 1 regarding Forefront TMG user authentication on AD mode, this part would look into URL content filtering feature. By default when URL filter is enabled during the startup wizard, TMG will block access to certain undesirable sites, including porno, gambling, malware etc.

Quite often, when you visit free news sites, you would be greeted with anonying Web Advertisements (Web Ads). And you wish to block them. Let's take a popular free news site at http://www.channelnewsasia.com/. Immediately, you will be seeing Web Ads on its top banner:

You can configure TMG to add Web Ads URL category into the block list. On the TMG console, right click on the "Web Access Policy" and choose "Configure Web Access Policy" on the context menu.

When the wizard starts, choose "No, do not create the rule for me" on the first page. On the next page (Blocked Web Destinations), click "Add", expand the "URL categories". Add "Web Ads" item. 

Complete the rest of the wizard as in default options. Apply the new changes. When you restart the client Web browser and visit the same Web site again, you'll notice that most (if not all) of the Web ads are gone.

New Site Categories Reporting in Service Pack 1

Among the new features in Service Pack 1, the new User Activity report displays the sites and site categories accessed by any user. All Forefront TMG reports also have a new look and feel. Reports can be generated one-time or recurrently. For example, you can see a sample one-time generated report for top visited URL categories as follows:

Rooms for Improvement in Reporting
Nevertheless, I still find the reporting features quite basic and lacking, as I couldn't find any filtering and search options in the report generation. 

Part 1: User Authentication for Forefront Threat Management Gateway (TMG) 2010

Traditional hardware-based network firewalls have one serious limitation - you can't control and track user access unless you are forcing them to perform another layer of firewall login on top of domain desktop login. If you really do, it certainly would cause much unhappiness among your users. On the other hand, if you can't control user access and track them directly from your firewall logs, you won't impress your auditors either. Hence, a simple solution to please both groups is to implement an Active Directory (AD) aware firewall solution.

One such firewall product to be seriously considered is from the maker of AD - Microsoft Forefront Threat Management Gateway (TMG) 2010 with latest Service Pack 1 (SP1). Besides being an AD-integrated firewall, it is also featuring Intrusion Prevention System (IPS), URL content filter, Web content cache and forward/reverse application proxies  - all rolled into one system.

Prior to deployment, various types of user authentication and application supports should be considered. There are 3 types of client setup to be considered:

1. SecureNAT clients that "hide" behind some IP address. Mainly used only for anonymous Internet access. However, it does not support user authentication.
2. Firewall clients (a.k.a TMG clients) that provide proxied winsock connections between the user applications and TMG. It automatically send client credentials with requests, which include Integrated AD authentication,  LDAP authentication, or RSA authentication (OTP).
3. Web Proxy clients supports above authentication mechanisms. However, the only application supported is the Web browser itself. No credentials are supplied if anonymous access is enabled.

From above, you can see that the strongest & most powerful client protection is provided by the TMG clients.

Forefront TMG Installation
The initial TMG setup can be pretty straightforward, as you will be guided by a step-by-step GUI wizard (click here for details). I have configured a test setup using Hyper-V as follows:

Mass Installation of TMG Clients using AD Deployment

To simulate mass deployment on an AD domain, I have assigned the TMG clients (a.ka. software pushdown) to all computers (or users) using Group Policy - Software Installation. For mass deployment, it is also more efficient to use client auto-detection and auto-configuration using information stored in Active Directory or DNS/DHCP (click here for details). AD detection would be preferred. In the absence of AD, DNS/DHCP discovery will be used.

To publish TMG information on AD, you would also need the ADconfig Pack, which can be downloded. To store information on Active Directory, at the command prompt, type: TmgAdConfig.exe add -default -type winsock -url [-f] where the service-url entry should be in the format http://{TMGServer}:8080/wspad.dat.

TMG Client Default Automatic Detection

Enable AutoDiscovery on TMG Management Console

(Networking -> Edit Internal Network -> AutoDiscovery Tab)

DNS Setup

Another important setup to note is the DNS configuration. Just as in normal AD environment, you should configure the TMG server and the clients to reference AD-integrated DNS servers (typically Domain Controllers). Both DC1 and client1 should set TMG server as default gateway. TMG server should have a default route set directly to the Internet.

For testing purposes, you may configure the default forwaders of DNS servers to your ISP name servers.

Firewall Policy

For user authentication testing for Internet access, I have configured the following firewall rules:

There are only 3 rules configured. The first rule is system default generated rule that block users from accessing potentially harmful sites. The second rule is to permit DC1 to forward client DNS requests to ISP name servers. The third rule is to limit HTTP/HTTPS access just to AD users authorised to access the Internet.

Client Testing

To simulate an unauthenticated user trying to access Internet, I login to Win7 client1 using local computer account. As expected, the Internet access is denied for anonymous access as shown under the "Logs and Reports" section of the TMG management console.

Subsequently, I login using a domain user account. Internet access to xin.msn.com succeeded.

Security Considerations for Active Directory Forest

As you can see from above, Forefront TMG succeeded in controlling user access based on Active Directory information. The logs also reveal User Identity and destination URLs instead of merely just client and desintation IP addresses as in most network firewall logs. As the firewall is relying on AD for user authentication and authorization, Microsoft advises further protection using forest segregation and one-way trust in Technet. Below is an extract:

In a domain environment, if Active Directory Domain Services (AD DS) is compromised for example by an internal attack, the firewall could also be compromised because a user with Domain Administrator rights can administer every domain member, including the server running Forefront TMG. Similarly, if the firewall is compromised, the domain in which Forefront TMG is located is also at risk. By default, the Domain Admins group is in the Administrators group on the Forefront TMG server.

At the edge, you can install Forefront TMG as a domain member or in workgroup mode. As a domain member, it is recommended that you install Forefront TMG in a separate forest (rather than in the internal forest of your corporate network), with a one-way trust to the corporate forest. This may prevent the internal forest from being compromised, even if an attack is mounted on the forest of the Forefront TMG computer. However, there are some limitations with this deployment; for example, you can configure client certificate authentication only for users defined in the Forefront TMG domain, and not for users in the corporate internal domain or forest.

Part 2: Image capturing & deployment using WDS

In earlier part 1, I mentioned using WinPE to capture Windows image files for further deployment. Another more scalable way is to use Windows Deployment Services (WDS) to do the same job. Instead of using CD boot, you can do network PXE boot with DHCP. There are several good resources online that outlined the step-by-step, so I won't re-invent the wheels. I would just summarize the steps and the url links to find them. Before that, it is assumed that you have already "sysprep-ed" the Windows host to be captured and put it to shut-off mode.

1. Setup WDS infrastructure by installing WDS server role. Other prerequisites include Active Directory infrastructure and DHCP.
2. Add Boot Image (boot.wim) to WDS server. Create and add a capture image by right-clicking on the selected boot image.
3. PXE boot the target system. From the start-up menu, choose to boot from the capture image.
4. Capture the target image on volume. Upload the new image to WDS.
5. You may now add the newly captured image as install image for subsequent Windows deployment.
6. For detailed steps from step 2 to 6, click here.

Load Sharing in MPLS VPN with Route Reflector

No matter what I did according to this Cisco article, I simply can't do load-sharing among the PE routers within our enterprise MPLS VPN. The central Route Reflector (RR) simply refuses to send more than 1 identical route, even though I have multiple PE routers attached to that route.

Subsequently, I come across this excellent MPLS VPN blog. It explains that there is no way to change the route reflector behavior.  The workaround is to simply assign different Route Distinguisher (RD) values for the same VRF on different routers. To enable route propagation, ensure that you export and import common Route Target (RT) across the PE routes. Finally, enable "maximum-paths eibgp " on each vpnv4 address family under the router bgp process, so as to inject more than one bgp route into the routing table.

And it works magic!

Example configuration according to above diagram

On Router PE A

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE B

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE C

ip vrf VRF_A

  rd 65001:101

 route-target both 65001:100

  .....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

On Router PE D

ip vrf VRF_A

  rd 65001:102

 route-target both 65001:100

.....

router bgp 65001

  address-family ipv4 vrf VRF_A

  maximum-paths eibgp 4

New Dynamic Memory in Hyper-V

The latest Service Pack 1 of Windows 2008 R2 offers a new Hyper-V feature known as Dynamic Memory. Some bloggers treat it like the "Memory Overcommitment" of VMWare where one may allocate more memory to all guest VMs than the actual total physical memory of the underlying host. I don't quite agree with this view. Unlike VMWare where VMs can still start up even though the total assigned VM memory exceeds the physical memory, Hyper-V won't allow the VM to start when the allocated base or startup memory exceeds the available memory left in the host.

Instead, Hyper-V treats memory as a shared resource that can be reallocated automatically among running virtual machines. Dynamic Memory adjusts the amount of memory available to a virtual machine, based on changes in memory demand and values that you specify. In dynamic memory, you assign 4 memory values to each VM:

1. Startup RAM: Specifies the required memory to start the VM.
2. Maximum RAM: Limits the maximum amount memory that the VM can be allocated.
3. Memory Buffer: Specifies how much memory Hyper-V would assign to the VM compared to the amount of memory actually needed. Memory buffer is specified as a percentage in relation to the actual needed memory. For example, if the memory committed is 1000 MB and the memory buffer is 20%, Hyper-V will attempt to allocate an additional 20% (200 MB) for a total of 1200 MB of physical memory allocated to the virtual machine. However, do note that this buffer is not maintained when there is not enough physical memory available in the host. 
4. Memory Weight: Determine the priority of individual VMs to be distributed with additional amount of memory when there are contention and insufficient memory available. This is the only memory value that can be adjusted dynamically when the VM is running.

Configuring Dynamic Memory

Step 1: First, you must upgrade the Hyper-V server to Windows Server 2008 R2 SP1.

Step 2: Either upgrade the guest VMs to the latest Service Pack e.g. Windows Server 2008 SP2 and Windows Server 2003 R2 SP2; OR install the SP2 version of Hyper-V integration services by clicking Insert Integration Services Setup Disk from the Action menu of Virtual Machine Connection. The former approach is recommended by Microsoft.

Step 3: Enable dynamic memory by going to the Memory Setting of the VM. Configure the startup memory  and maximum memory. To do so, the VM must be in Off state.

Step 4a: Observe the memory in real time. When you start the VM, you would see the following VM status info on Hyper-V manager:

• Assigned Memory shows the amount of memory allocated to the VM at this time.
• Memory Demand shows how much memory the VM needs at this time, which is based on total committed memory obtained from the performance counter in the VM.
• Memory Status shows how much of the buffer amount specified for the virtual machine is available. OK indicates that there is enough physical memory available to give the virtual machine the full amount of memory buffer. Low indicates that Hyper-V does not have sufficient memory to meet the full memory buffer requirement. Warning indicates that no more memory buffer can be distributed to the virtual machine. 

Step 4b: You can also observe the new Hyper-V Dynamic Memory counter added in Performance Monitor in the Hyper-V host.

For further troubleshooting details, refer the the Microsoft Technet on Dynamic Memory.