Remote EFS for Cross-Forest AD Users

I was working on a project that allows encrypting files on remote file share, which was not enabled by default. I've to setup the pre-requite infrastructure, including fully functional Active Directory, Public Key Infrastructure, roaming user profiles (RUP) and the file shares. Group policies would have to be configured to make everything happens. In addition, users from trusted forest must also be allowed to login to the resource forest, and be enrolled with EFS certificate to perform the same remote file encryption. I find there is a lack of clear and cohesive documentation regarding cross-forest support for remote EFS on Microsoft Technet. Here, I would attempt to document down what I did to make cross-forest for remote EFS operations work. Meanwhile, I would also thank the excellent Microsoft premium support team for providing me the necessary insights over multiple email to get it up and running.

Let's get started. First thing first, before performing cross-forest operations, you have to ensure same forest EFS operations work.

Understanding Remote EFS

Local EFS is pretty straightforward: encrypting file on local file-systems. Remote EFS comes in 2 versions: (1) remote encryption/decryption with SMB file share and (2) remote encryption/decryption with WebDAV (a.k.a Web folders). 

For (1) SMB mode, roaming user profiles for EFS certificate must be enabled, so that the file share server may impersonate the users and load the EFS keys from roaming user profiles to perform encryption/decryption on users' behalf. To enable impersonation, the file server must be trusted for delegation. As the file is decrypted when it leaves the server, the data across the network is left in plain-text. Hence, you'll have to deploy additional IPSec or SSL to protect the transport path. 

As for (2) WebDAV mode, the file is automatically copied from the Web folder to the user’s computer, encrypted on the user’s computer, and then returned to the Web folder over HTTPS. The advantage is that the computer hosting the Web folder does not need to be trusted for delegation and does not require roaming or remote user profiles. However, it comes with a serious limitation. I've tested it to be rather "slow" in access performance and Microsoft recommends that the file size should be kept below 60MB.

Refer to this Technet link on Remote EFS Operations for further information. In this post, I would focus solely on (1) Remote EFS with SMB file share. 

Phase 1: Remote EFS on SMB file share within Forest

• Provision network share and enable roaming user profiles for users. See "Deploy Roaming User Profiles"
• Setup Active Directory Certificate Services and configure EFS certificate auto-enrollment for users. See "AD CS and PKI Step-by-Steps, Labs, Walkthroughs, HowTo, and Examples"
• Login with a test user account on a client machine. Verify that an EFS certificate has been auto-enrolled using "certmgr.msc". Logout the user, so that the roaming cert can be synced to the RUP.
• Login again with the same user account. Verify that the EFS cert has been synced to the RUP folders. See this roaming cert troubleshooting guide
• Setup a file server and provision a network share for remote EFS. Delegate the file server to be trusted. See "Remote Storage of Encrypted Files Using SMB File Shares and WebDAV"
• Map the network drive on the client machine. Try to create some files and encrypt them. Verify that the files are encrypted with the same certs on your profiles. Otherwise, the file server might be requesting a new EFS cert on your behalf instead of loading it from RUP. If the files fail to encrypt,  it could due to different roaming profile folder structures created by different Windows client version. See this troubleshooting guide.

Phase 2: Supporting cross-forest users for certificate auto-enrollment and remote EFS

• Setup two-way forest trust. In this context, this forest (hosting resources, such as file servers etc)  is known as "resource forest". The forest where users come from is known as "account forest". Create new security group with "Domain Local" scope and add cross-forest users to this group.
• Allow "Read" and "Auto-enrollment" for the cross-forest group to the EFS user template
• Configure Cross forest certificate enrollment. Run the "PKIsync.ps1" script to sync PKI objects, which is critical to facilitate cross-forest enrollment. Run the command on the account forest with "Enterprise Admin" rights specifying the source and target forests. 
• The Technet link for cross-forest enrollment above missed out the step of copying the pkiEnrollmentService object over (i.e. certutil -config can't be used). Use this command instead:

• PKISync.ps1 -sourceforest source.local -targetforest target.local -type CA -cn "CA-Name"

• To verify the cert templates have been copied over, use the "AD Sites and Services" on the target DC, click "View" and tick "Show Service Node". Look under the folder "Services\Public Key Services\Certificate Template" to check the templates.
• Add the Issuing CA server to the "Cert Publishers" Domain Local group on the account forest. 
• Enable LDAP referral support on enterprise CAs: certutil -setreg Policy\EditFlags +EDITF_ENABLELDAPREFERRALS
• Deploy group policy to enable "Allow Cross-Forest User Policy and Roaming User Profiles" and set "roaming profile path for all users logging onto this computer" on the client machines for cross-forest users, as well as the file servers as shown below:

• User group policy would not take effect for cross-forest user logon to domain computer. To configure user policy, configure loopback policy to facilitate auto-enrollment on client computers logon by cross-forest users  Even if you don't enable it explicitly, loopback with replace mode would still be enabled for cross-forest user login by default. Enable Auto-enrollment on user configuration. Leave the "Certificate Enrollment Policy" as "Not configured".

• Test domain logon using a cross-forest user account. Perform the same verification procedures as earlier mentioned for single forest operations.
• If remote EFS operations fail, despite auto-enrollment works and the RUP certs are synced, it might be due to the group policy fail to take effect on the file server. The server attempts to impersonate a cross-forest user but is disallowed by default as shown on event viewer below. Furthermore, the server is not aware of the location of RUP folders for cross-forest users. 

• Apply gpupdate and gpresult /h gpresult.html on the file server. Ensure "Allow Cross-Forest User Policy and Roaming User Profiles" and "roaming profile path for all users logging onto this computer" are applied 
• Cross-forest users should be able to perform remote EFS operations on this resource forest.

Modifying Built-in AD Objects

While working on cross-forest cert enrollment, I noticed I couldn't add an cross-forest object to an AD built-in group object, e.g. "Cert Publishers" group. The built-in group is of "Global Security" scope that is unable to add objects outside a forest. Rightfully, it should be of "Domain Local" type, so that cross-forest objects can be added. If it's not a built-in object, we can simply change it to "Universal" and then "Domain Local" using "AD users and computers" console. 

Nevertheless, we can use the SYSTEM account to modify the built-in group scope.

1. Download pstools from link: http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx.
2. Extract it and get psexec.exe and copy it to a Domain Controller (DC).
3. Open new command and run the command psexec.exe -s “cmd”, click Accept/Yes.
4. We will get a new command that run under System account.
5. Under new command prompt, please run the command below:

                dsmod group “CN=Cert Publishers,CN=Users,DC=contoso,DC=com” -scope U
                dsmod group “CN=Cert Publishers,CN=Users,DC=contoso,DC=com” -scope L

This would convert built-in group scope to Domain Local.

Note: If the AD was first created on Windows Server 2000, the group scope type was created "Global Security" and would remain as it was, even when there was AD prep and upgrade subsequently.

Verifying SIDHistory of user accounts

When you're performing user and resource migration between forests with trust, it's important to enable SIDHistory and disable SID quarantine during the migration process. As migrated users on target forest will get new domain SIDs, it's important for them to retain the old SIDs in their originating forest. This is to ensure that the migrated users on new target forest are still able to access resources in the source forest until the migration is complete.  This blog post sums it up using Active Directory Migration Tool (ADMT).

But how to verify a SID history of the migrated user is of the same SID in the source forest?

On new target forest:

dsquery * -Filter "(samaccountname=userid)" -attr sidHistory

On source forest:

dsquery * -Filter "(samaccountname=userid)" -attr objectSid

Compare the SID values of both output and ensure they are the same.

Setting the ACL for Home and Roaming Profiles

Wonder how should you set the ACL of Share and NTFS of the network share for Users' Home folders and Roaming Profiles? Check out this Technet Blog: Automatic creation of user folders for home, roaming profile and redirected folders

By default, all newly created folders are set with inheritable permissions that include Read permission for all users. As a result, users would be able to see all other users' home folders. Access Based Enumeration (ABE) is designed to prevent users from viewing other folders that they have no read access. It can be easily enabled on the "Share and Storage Management" console. However, inheritable permission get in the way because it permits all users to have "Read" access to all folders.

For ABE to work, you'll have to remove that inheritable permissions after the users' home folders are automatically created. You can have a Powershell Script that take in CSV file (exported by csvde) and remove all inheritable permissions on the user home folders. And this is my script:

import-csv C:\temp\users.csv | foreach-object {
  # individual user name
  $user = $_.sAMAccountName
  # user home folder
  $newPath = Join-Path "\\FileShare\Home$" -ChildPath $user
  $acl = Get-Acl $newPath
  # this would remove inheritable permission
  $acl.SetAccessRuleProtection($true,$false)
  # additional custom permission added (optional)
  $permission = "MyDomain\$user","Modify","Allow"
  $accessRule = New-Object System.Security.AccessControl.FileSystemAccessRule $permission
  $acl.SetAccessRule($accessRule)
  $acl | Set-Acl $newPath
}

If you happen to encounter situation whereby you can't move or remove the user profile folders, you'll have to take ownership of the folder recursively. Here're the command lines:
takeown /F folder-name /R /D y 
icacls folder-name /grant administrators:F /T

New Storage Spaces, SMB3.0 and SOFS in Windows Server 2012

One big game-changing difference in Windows Server 2012 is the new Storage Spaces feature and SMB3.0. Storage Spaces organizes a bunch of hard-disks neatly into virtual storage pools that can be easily expanded by simply adding more hard-disks. Virtual storage pool can also be added to support Clustered Storage Space for fail-over clustering (see: "How to Configure a Clustered Storage Space in Windows Server 2012"). Minimally, each node needs to have direct access to shared disk resources (at least 3 x SAS drives) i.e. SAS JBOD with no RAID sub-system, such as using PCIe non-RAID HBA connection.

SMB3.0 provides many enhancements for improved performance, resiliency and security, including:

1. SMB Scale-Out: transparently redirect SMB client connections to a different file server cluster node.  
2. SMB Direct (SMB over RDMA): enables direct memory-to-memory data transfers between servers, with minimal CPU utilization and low latency, using standard RDMA-capable network adapters (iWARP, InfiniBand, and RoCE). Any application which accesses files over SMB can transparently benefit from SMB Direct.
3. SMB Multichannel – takes advantage of multiple network interfaces to provide both high performance through bandwidth aggregation, and network fault tolerance through the use of multiple network paths to data on an SMB share.  Fast data transfers and network fault tolerance.
4. Transparent Failover and node fault tolerance – Supporting business critical server application workloads requires the connection to the storage back end to be continuously available. The new SMB server and client cooperate to make failover of file server cluster nodes transparent to applications, for all file operations, and for both planned cluster resource moves and unplanned node failures.
5. Secure data transfer with SMB encryption – protects data in-transit from eavesdropping and tampering attacks. Encrypting File System (EFS) is still required to protect data at rest though.

Reference: "SMB 2.2 is now SMB 3.0".

You can enable cluster disk as new Cluster Shared Volume (CSV). CSV enables all cluster nodes to "own" a "shared" volume at the same time i.e. Active-Active configuration. When CSV 1.0 was first introduced in W2K8 R2, it was only meant for Hyper-V storage to support Live Migration. Match the new CSV 2.0 storage with SMB3.0, it provides a real solid NAS-based alternative to SAN for performance and resiliency at a better value known as Scale-Out File Server (SOFS). Meanwhile, Microsoft is working with hardware partners to create a cluster-in-a-box (CiB) architecture if you prefer appliance-based SOFS solution. 

SOFS can be used as file-based storage spaces for Hyper-V and MS SQL clusters over SMB 3.0. Without expensive SAN storage (replaced by SOFS using shared SAS JBOD) in the picture, the new Hyper-V and SQL cluster would look like below (taken from TechEd 2012):

Subsequently, I did a quick test on SOFS using iSCSI storage. Even without fail-over clustering, "Share Nothing Live Migration" is also possible for non-clustered Hyper-V hosts using Hyper-V replica.

Having singing much praises to SOFS, do note that it is still not meant for every situation. Microsoft recommends that SOFS should not be used if your workload generates a high number of metadata operations, such as opening files, closing files, creating new files, or renaming existing files, which is typical for end-user file shares. Microsoft publishes the following chart to help you to decide when to use traditional file share and SOFS (taken from "When to use Scale-Out File Server"):

If you're already running 10Gigabit Ethernet or higher in your data center, you should further optimise your existing investment and leverage on the full performance benefits of SMB Direct (i.e. SMB over RDMA). Do note that the servers should have Network Interface Cards (NICs) that support RDMA (iWARP or RoCE). Here is a link on RDMA enabled NICs that support Windows Server 2012.

I've also come across this informative MVP blog about the new SOFS that can potentially replace SAN-based solution for server clustering. Here's the extract:

Scale Out File Server (SOFS)

Normally we want our storage to be fault tolerant. That’s because all of our VMs are probably on that single SAN (yes, some have the scale and budget for spanning SANs but that’s a whole different breed of organisation).  Normally we would need a SAN made up fault tolerant disk tray$, switche$, controller$, hot $pare disk$, and $o on. I think you get the point. Thanks to the innovations of Windows Server 2012, we’re going to get a whole new type of fault tolerant storage called a SOFS.

When I’ve talked about SOFS many have jumped immediately to think that it was only for small businesses.  Oh you fools!  Never assume!  Yes, SOFS can be for the small business (more later).  But where this really adds value is that larger business that feels like they are held hostage by their SAN vendors.  Organisations are facing a real storage challenge today.  SANs are not getting cheaper, and the storage scale requirements are rocketing.  SOFS offers a new alternative.  For a company that requires certain hardware functions of a SAN (such as replication) then SOFS offers an alternative tier of storage.  For a hosting company where every penny spent is a penny that makes them more expensive in the yes of their customers, SOFS is a fantastic way to provide economic, highly performing, scalable, fault tolerant storage for virtual machine hosting.

Authenticating SMTP users on Exchange Edge

I thought of authenticating all POP3/SMTP external users. POP3 access are provided by Client Access Server (CAS) and is joined to domain. No problem in authentication for POP access.

As for SMTP service, it is only provided by Hub Transport or Edge Transport. Hence, it made much security sense that I created a new Receive Connector on the edge server and enable "Basic Authentication". But when I configured Outlook client for SMTP authentication, the edge server rejected the authentication. Initially, I thought it could be due to Exchange ACL error or AD LDS faults within the Edge server. I came to realize that this is a wrong concept when I come to this Technet blog. Remember that AD LDS is for extending AD partition to the perimeter network and it's not meant for authentication (only a full Domain Controller or RODC does authentication but Exchange doesn't support the latter). 

Important note:

Configuring SMTP

Most commonly, however, your clients will be authenticating for the purposes of identifying themselves (sender permissions checks) and prove that they are allowed to relay. This authorization can be done by Edge only if it is in the domain. Since is not be the most common configuration, the Hub role may be more suited for this purpose. 

Hyper-V vs. vSphere: Understanding the Differences

SolarWinds did a very good job at comparing Hyper-V and vSphere. The views are unbiased and independent. The upcoming Hyper-V 3.0 in Windows Server 2012 are also briefly covered. Here're the links:

1. Webcast
2. WhitePaper 

Microsoft also did a comparison (of course - from Microsoft's perspectives): 

• Why Hyper-V? Competitive Advantages of Windows Server 2012 Release Candidate Hyper-V over VMware vSphere 5.0

And so did VMWare:

• Top 10 Reasons Why VMware vSphere 5 is Years Ahead of the Competition

Both Microsoft and VMWare did the comparison by highlighting their "strengths" and their competitor's "weaknesses". Hence, you can also get a balanced view by reading both whitepapers side-by-side. In my personal opinions, it's true that vSphere is still heading way ahead of Hyper-V R2. Hyper-V 3.0 will narrow the gaps significantly and offered even better than "good-enough" features for most enterprises. Coupled with "irresistible" unlimited "free" VM rights from Hyper-V Data-centre edition and hearing no further new revolutionary announcements from VMWare, it seems to me that VMWare might be fighting a losing head-to-head battles against the Redmond software giant.