New Storage Spaces, SMB3.0 and SOFS in Windows Server 2012

One big game-changing difference in Windows Server 2012 is the new Storage Spaces feature and SMB3.0. Storage Spaces organizes a bunch of hard-disks neatly into virtual storage pools that can be easily expanded by simply adding more hard-disks. Virtual storage pool can also be added to support Clustered Storage Space for fail-over clustering (see: "How to Configure a Clustered Storage Space in Windows Server 2012"). Minimally, each node needs to have direct access to shared disk resources (at least 3 x SAS drives) i.e. SAS JBOD with no RAID sub-system, such as using PCIe non-RAID HBA connection.

SMB3.0 provides many enhancements for improved performance, resiliency and security, including:

1. SMB Scale-Out: transparently redirect SMB client connections to a different file server cluster node.  
2. SMB Direct (SMB over RDMA): enables direct memory-to-memory data transfers between servers, with minimal CPU utilization and low latency, using standard RDMA-capable network adapters (iWARP, InfiniBand, and RoCE). Any application which accesses files over SMB can transparently benefit from SMB Direct.
3. SMB Multichannel – takes advantage of multiple network interfaces to provide both high performance through bandwidth aggregation, and network fault tolerance through the use of multiple network paths to data on an SMB share.  Fast data transfers and network fault tolerance.
4. Transparent Failover and node fault tolerance – Supporting business critical server application workloads requires the connection to the storage back end to be continuously available. The new SMB server and client cooperate to make failover of file server cluster nodes transparent to applications, for all file operations, and for both planned cluster resource moves and unplanned node failures.
5. Secure data transfer with SMB encryption – protects data in-transit from eavesdropping and tampering attacks. Encrypting File System (EFS) is still required to protect data at rest though.

Reference: "SMB 2.2 is now SMB 3.0".

You can enable cluster disk as new Cluster Shared Volume (CSV). CSV enables all cluster nodes to "own" a "shared" volume at the same time i.e. Active-Active configuration. When CSV 1.0 was first introduced in W2K8 R2, it was only meant for Hyper-V storage to support Live Migration. Match the new CSV 2.0 storage with SMB3.0, it provides a real solid NAS-based alternative to SAN for performance and resiliency at a better value known as Scale-Out File Server (SOFS). Meanwhile, Microsoft is working with hardware partners to create a cluster-in-a-box (CiB) architecture if you prefer appliance-based SOFS solution. 

SOFS can be used as file-based storage spaces for Hyper-V and MS SQL clusters over SMB 3.0. Without expensive SAN storage (replaced by SOFS using shared SAS JBOD) in the picture, the new Hyper-V and SQL cluster would look like below (taken from TechEd 2012):

Subsequently, I did a quick test on SOFS using iSCSI storage. Even without fail-over clustering, "Share Nothing Live Migration" is also possible for non-clustered Hyper-V hosts using Hyper-V replica.

Having singing much praises to SOFS, do note that it is still not meant for every situation. Microsoft recommends that SOFS should not be used if your workload generates a high number of metadata operations, such as opening files, closing files, creating new files, or renaming existing files, which is typical for end-user file shares. Microsoft publishes the following chart to help you to decide when to use traditional file share and SOFS (taken from "When to use Scale-Out File Server"):

If you're already running 10Gigabit Ethernet or higher in your data center, you should further optimise your existing investment and leverage on the full performance benefits of SMB Direct (i.e. SMB over RDMA). Do note that the servers should have Network Interface Cards (NICs) that support RDMA (iWARP or RoCE). Here is a link on RDMA enabled NICs that support Windows Server 2012.

I've also come across this informative MVP blog about the new SOFS that can potentially replace SAN-based solution for server clustering. Here's the extract:

Scale Out File Server (SOFS)

Normally we want our storage to be fault tolerant. That’s because all of our VMs are probably on that single SAN (yes, some have the scale and budget for spanning SANs but that’s a whole different breed of organisation).  Normally we would need a SAN made up fault tolerant disk tray$, switche$, controller$, hot $pare disk$, and $o on. I think you get the point. Thanks to the innovations of Windows Server 2012, we’re going to get a whole new type of fault tolerant storage called a SOFS.

When I’ve talked about SOFS many have jumped immediately to think that it was only for small businesses.  Oh you fools!  Never assume!  Yes, SOFS can be for the small business (more later).  But where this really adds value is that larger business that feels like they are held hostage by their SAN vendors.  Organisations are facing a real storage challenge today.  SANs are not getting cheaper, and the storage scale requirements are rocketing.  SOFS offers a new alternative.  For a company that requires certain hardware functions of a SAN (such as replication) then SOFS offers an alternative tier of storage.  For a hosting company where every penny spent is a penny that makes them more expensive in the yes of their customers, SOFS is a fantastic way to provide economic, highly performing, scalable, fault tolerant storage for virtual machine hosting.