Over the past few weeks, us geeks at Money Dashboard have been hard at work building our production environments, and as I am sure you can imagine we hit a few issues.
In order to help anyone else out who might be thinking about, or even is deploying a fairly complex environment around Hyper-V I thought I would share our findings.
First off, as I’m sure you can all appreciate, I am unable to go into any real specific details on our implementation, so some of this information maybe be a little strange, or difficult to follow – bear with me and hopefully if you ever find yourself in a similar situation it might just help you out!
iSCSI and Virtual Machines
If, like us, you are virtualising your SQL Server’s, then do not forget that you will need to bring some iSCSI (or whatever storage system) mounts through to the VM’s. This on the surface does not pose a problem, but we DID hit problems when pulling iSCSI through. As you can imagine, we are running Jumbo Frames (MTU 9000) on our iSCSI network in order to optimise throughput, but the default VM adapters only support standard packet sizes (i.e. MTU 1500). In order to get around this, you need to use the Synthetic Network Adapter in the Hyper-V VM, and be sure to set the properties to enable Jumbo Frames. You must also have the physical nic on the server set for Jumbo Frames too. Always worth checking with the following command:
netsh interface ipv4 show subinterface
You may, like us, then notice some packet loss on the iSCSI Adapter. In our case this turned out to be something strange going on with the way the Synthetic adapters were behaving with our Broadcom nics (BCM5709 in case anyone is interested!). Disabling all the offload components (TCP, iSCSI and Checksum) fixed the problem, but we still do not know exactly why this was occurring …
QoS …. do NOT forget it
Make sure you split your Live Migration and Heartbeat traffic onto separate networks, and oh most certainly remember to apply those Quality of Service rules on the switchgear.
We forgot, and as soon as Live Migration kicked off, the complete Hyper-V cluster went mental … it thought that all the other nodes had failed, so EVERY node went to start EVERY virtual machine. As you can no doubt imagine, absolute chaos ensued, and the virtual machine disks were corrupted (one catch with using the new Cluster Shared Volumes it seems).
Dell EqualLogic, and BACS
We are lucky enough to be using pretty much all Dell kit, including the Dell EqualLogic PS6000XV as our SAN. One snag we did hit is that you really do not want to route your iSCSI traffic over a virtual adapter created through the Broadcom BACS suite … on the surface it will appear to work, but when you start looking at it carefully you will notice that there appears to be a significant bandwidth limitation creeping in somewhere. Not sure if it was the BACS drivers, or the Dell MPIO driver, but it disappeared when we reverted to use proper physical NICS. Equally, do not forget to install the Dell MPIO drivers into any virtual machines that are using the iSCSI :)
Adding Hyper-V Clusters in Microsoft System Center Virtual Machine Manager
When you are finally ready to add your machines into SCVMM, add the cluster name – not the individual machine names. It seems that if you add the individual machines, SCVMM does NOT treat them as an HA cluster. I haven’t found any logical way of merging multiple machines into a cluster in SCVMM, or any real reason why it doesn't prevent you from adding the individual nodes anyway (it can see they are in a HA cluster configuration after all).
Summary of the kit we used:
Dell EqualLogic PS6000XV SAN
Dell PowerConnect 6248 gigabit switch stack
Dell R200 1U Rack Mount Servers
Dell M1000e Blade Centre
An awful lot of cabling.
Microsoft Windows Server 2008 R2 Enterprise (Both full and Core)
Microsoft Windows Server 2008 R2 Standard (Both full and Core)
Microsoft System Center Virtual Manager Manager 2008 R2
Microsoft System Center Operations Manager 2007 R2
And a ton of custom scripts.
Thanks go to Dave Veitch from Company Net for assisting in the configuration!