New Server Form Factor: Cluster-in-a-Box

Traditional server clustering has always been one of the main 'bottlenecks' and complaints for most IT project roll-outs. The most common excuses from server admins: too complex, too expensive, taking too much power, rack space, air-con and the list goes on. To address these issues, there is a new breed of servers - not 1U or 2U rack mountable server and monstrous server blade - it's "Cluster-in-a-Box" or CiB in short. 

Simply put, all server clustering components, including computing, storage, clustering software and network, are pre-fabricated and pre-installed in a single box when deliver to your data centers. You don't have to worry about what kind of detailed specifications that you should put in your procurement tenders. Most importantly, most products claim to save power, cooling, and rack space. Take Nutanix for example. It claims to do away with expensive SAN storage and simplify cluster setup in 30 min. Furthermore, a 4-node cluster in a 2U rack mountable chassis. Compare Nutanix 1450 against traditional server clustering, the Dell equivalent would look something like:

1. 4 x Dell R610 servers
• 2 x Intel 6-core Xeon E5-2620
• 64GB RAM
• 400GB SSD
• 2 x GE and 2 x 10GE
• 2 x HBA controllers
2. 2 x FC SAN switches
3. 1 x FC SAN storage 
• 16 x 1 TB SATA HDD

Adding up all components would take up about 8U of rack spaces with 7 different boxes or about 400% in saving in rack spaces against a single 2U Nutanix box.

Of course, traditional server vendors are not sitting back. Now, we are seeing new server product offerings like:

• Dell Power Edge VRTX
• Fujitsu’s PRIMERGY
• HP Cluster Platform Express

Look like we are entering the next wave of data center revolution.

Does DELL MD32xx support hardware VSS snapshot for DPM Backup?

Answer: Yes, but it's not official. It's best effort (due to its low cost as compared to its high cost Compellent counterpart).

Hardware VSS snapshot provides an efficient way for DPM to backup Cluster Shared Volumes (CSVs) on Hyper-V cluster. Without hardware assisted snapshot, the CSV access would be placed under Redirect I/O mode as long as the backup process is taking place (which can be a long process). This can significantly reduce the I/O performance for other VMs on the same LUN. That's why it's more efficient to keep as small number of VMs per CSV as possible for software VSS. With hardware assisted VSS, a temporary snapshot is instead created on the free space on the SAN volume. CSV would only be placed on Redirect mode for a couple of minutes.

Screenshot of VSS Hardware Snapshot (new virtual disk temporarily created denoted with a clock icon) on MD Storage Manager during DPM backup:

How do we configure VSS setting on Dell MD storage?

Step 1: Install the VSS provider given on the Dell MD Storage Manager DVD or ISO image.

Step 2: Confirm Dell hardware VSS provider is installed on the system
C:\> vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Provider name: 'SmVssProvider'
   Provider type: Hardware
   Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
   Version: 10.84.02

Step 3: Configure the default VSS behavior on the storage as shown below. Run "C:\Program Files (x86)\Dell\SMprovider\provider\SmRegTool.exe" on the console installed with Dell Storage Manager. Connect to the storage using OOB Management IP addresses.

Insufficient counts after KMS host migration

For this new AD forest, I've been leveraging on someone else KMS service. As it grows bigger, time to setup its own KMS host. It's pretty straightforward to migrate KMS server. Plenty of step-by-step online like this. I have also manually updated the _vlmcs._tcp SRV record on DNS server.  And there is a problem. If you attempt to activate small number of new KMS clients after migration, it would report insufficient count. For KMS to work, you would need at least either 5 KMS servers or 25 KMS clients.  

So, I tried to re-activate existing machines by uninstalling existing KMS client key and re-installing it. The count level on the KMS host still won't jump and remain under 5. You can see the current count level by entering "slmgr -dlv" on the KMS host.

After going through all the slmgr options, I noticed an option to disable KMS host caching. Could it because the existing machines still activate on old KMS server? 

On a number of existing KMS client machines:

> slmgr -ckhc

> slmgr -ato

Hurry, the count is raised! New KMS clients are activated at once.

How to delete Virtual Disk in Dell MD during Operation?

How to delete Virtual Disk in Dell MD during operations? Answer is you can't, you've to wait for the operation to complete.

That sounds fair but not when you've waited several hours (if not days) for the operation to complete. Power cycle the MD storage still can't abort the operation. From the DellMD manager console, you've absolutely no idea to know when it would ever complete.

The next best news is that you can run a command utility "smcli.exe" to roughly gauge the progress. This is typically found under  "C:\Program Files (x86)\Dell\MD Storage Manager\client" folder.

> smcli -n ArrayName -c "show virtualdisk [\"VirtualDiskinOperation\"] actionprogress";

Performing syntax check...

Syntax check complete.

Executing script...

Virtual Disk xxx
Action: RAID Level Change
Percent Complete: 66%
Script execution complete.

SMcli completed successfully.

Unable to refresh a stuck VM in SCVMM 2012 console

If you are unable to refresh a stuck VM on SCVMM 2012 and you see the following error:

"VMM cannot find the Virtual hard disk object' error 801"

You may need to run a SQL script to remove the orphaned objects from the database by following the steps below.

1. Stop the System Center Virtual Machine Manager service on the VMM 2012 server.
2. Back up the Virtual Manager database.
3. Open SQL Management Studio and run the following script on the VMM database:

BEGIN TRANSACTION T1

DECLARE custom_cursor CURSOR FOR
SELECT VHDId, VDriveId from
dbo.tbl_WLC_VDrive WHERE [VHDId] NOT IN
(SELECT VHDId from dbo.tbl_WLC_VHD WHERE VHDID IS NOT NULL)

DECLARE @VHDId uniqueidentifier
DECLARE @VDriveId uniqueidentifier

OPEN custom_cursor
FETCH NEXT FROM custom_cursor INTO @VHDId, @VDriveId

WHILE(@@fetch_status = 0)
BEGIN
if(@VHDId is NOT NULL)
DELETE FROM dbo.tbl_WLC_VDrive
WHERE VDriveId = @VDriveId
FETCH NEXT FROM custom_cursor INTO @VHDId, @VDriveId
END
CLOSE custom_cursor
DEALLOCATE custom_cursor

COMMIT TRANSACTION T1

4. Start the System Center Virtual Machine Manager service again and refresh problem VMs. The VMs should return to a proper reporting state.

The above resolution is extracted from this post: Performing an operation in Virtual Machine Manager fails with error 801

If you encounter Error (2606),
"Unable to perform the job because one or more of the selected objects are locked by another job."

Run this command on the affected host to resync the performance counter: winmgmt /resyncperf

Hyper-V (WS2012) Backup on Guest VM (WS2008)

If you're running backup jobs (e.g. DPM2012) on WS2012 Hyper-V with older guest VMs (e.g. WS2008), you might encounter this error:

The replica of Microsoft Hyper-V \Backup Using Child Partition Snapshot\[VM] on [Host] is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106) 

To resolve it,
1) ensure that the host contains this update KB2770917 (available in Windows update)
2) Update the Integration services on the older guest VMs e.g. WS2008R2 etc

SQLPrepInstaller of DPM2012 installation using remote SQL server

This Technet link describes the setting up of remote SQL server of Microsoft's latest backup solution - System Center Data Protection Manager (DPM) 2012 SP1.

A missing step is not mentioned for using remote SQL instance (i.e. SQL not installed on same server as DPM). You'll have to run the "SQLPrepInstaller" on the SQL server. Insert the DVD, ISO or file share image of DPM installer on the SQL server. Click on the following menu on the launcher.