If you haven’t thought about taking advantage of virtualization to make clustering more cost effective, you might take some time to explore the idea. Clustering with VMs gives you versatility and lets you combine physical machines with VMs. For example, you can deploy clustering multiple VMs within a physical server, clustering VMs across physical servers, and using a VM as a cost-effective standby server.

Let’s look at these possibilities. Assume that the guest OS is Windows Server and the cluster software is Microsoft Clustering Service (MSCS). The two key requirements that need to be fulfilled for MSCS to function are having a shared disk and a network so that you can establish a heartbeat to ensure that your clustered machine is available.

On a physical server with the ESXi hypervisor, you can create a virtual shared disk and virtual network adaptors without needing special hardware or extra network adaptors.  You can cluster Windows Server VMs using MSCS and use your virtual shared disk as the shared disk for the cluster and the virtual network for the heartbeat. This type of cluster can deal with only software failure and does not protect from hardware failure; this type of cluster is typically meant used for development and testing.

If you have two physical servers with an ESXi hypervisor and physical shared storage, you can create virtual shared disks that are stored on a shared physical disk. MSCS using this shared virtual disk to cluster two Windows Server VMs on separate ESXi hosts. These types of clusters offer protection from both software and hardware failures. This configuration can be used to consolidate several physical server clusters. 

Virtualization can have a greater positive impact in a small business than it does in a larger business. It can get your existing applications off old hardware, provide some much-needed fault tolerance, and it can allow your IT capabilities to grow as the business grows. How cheaply can you get virtualization into your small business? Pretty cheaply, but you still must plan it out.

Since your small business probably doesn’t have any extra virtualization-capable servers just lying around, you must shell out the shekels for the hardware. And before you can do that, you must justify why. Here are a few scenarios:

• Adding a second Active Directory domain controller for your Small Business Server (SBS) domain. You must have a second DC, even if it’s just a VM you bring up occasionally on a notebook, in your domain. If you don’t and you have a serious problem with SBS, you may have irreparably damaged your business. This one act of AD fault tolerance alone can justify the hardware purchase.
• Retiring an old, out-of-support server by migrating its applications to a VM.
• Creating a test or staging VM for upgrades to your existing applications.
• Responding quickly to a business opportunity; for example, creating an externally facing Web server to provide a new service.

I’m sure you can think of a few more yourself. You can buy a very competent, virtualization-capable quad-core tower server with 16GB of RAM for $2000; that’s enough to host six or seven moderate-use VMs.

What about the hypervisor? Hyper-V Server 2008 R2 has every one of the features that the full Windows Server 2008 R2 Hyper-V product has, and it’s free. Its two caveats are that 1) you must manage it remotely, as you’d manage a Server Core installation, and 2) it doesn’t have any built-in licensing for its VMs as Windows Server 2008 Standard, Enterprise, or Datacenter does. You can work around the first restriction by using standardized remote management tools (such as the Remote Server Administration Tools) from your client or another server. If you have available licenses for Windows Server 2003 or 2008, you can use them to license your new VMs; otherwise it might be less expensive to purchase a copy of Windows Server 2008 R2.

Now you have your environment. How do you manage it? Use the native Hyper-V Manager utility. You don’t get the fancy bells and whistles that Virtual Machine Manager (VMM) provides, but as long as you have only one host server, you’ll be okay.

If you aren’t using VMM, how do you migrate your existing applications off that old server and into a VM on your host? Use Disk2vhd. This handy little tool will virtualize a physical volume to the .VHD format, which you can then import into Hyper-V.

Now that you’ve got your virtual infrastructure started, how do you protect it? Use your existing backup utilities to back up your VMs just as if they were physical machines. Another solution– and a good reason to purchase a copy of Windows Server 2008 R2 for the host– is to attach one or two big USB drives to the host, and use the built-in Windows Server Backup (WSB) to automatically back up the host and its VMs to the external drives. WSB will take care of the space management and versioning for you. And the host backup uses the Volume Shadow Service (VSS) to safely back up the VMs in a manner that they’re aware of, so their applications (like AD) will work correctly when restored. In case of a host failure, you could also use a virtualization-capable notebook (they’re becoming pretty common) as a hot spare device to temporarily restore one or two of your most important VMs until the host can be repaired. With Windows 7 installed on the notebook, you can use boot-from-VHD or even Windows Virtual PC to run the restored VMs.

With a little creative planning, you can get your small business into virtualization inexpensively and add capabilities and flexibility that might otherwise be out of reach.

Shorter is better. Simpler is faster. When you’re looking to get from Point A to Point B in the real world, if all other factors are equal the shortest route will be the fastest. This also applies in the computing world, but over time hardware and software engineers have come up with many ways to cheat on this maxim. Virtualization is one step removed from the shortest path – an OS directly on the native physical box – so its performance has always been a topic of interest. When you’re thinking about a Hyper-V implementation, what kind of performance optimization should you be doing?

There are four major subsystems to focus on in any discussion of computer performance tuning: processor, networking, memory, and disk I/O. I have first-hand experience with VM disk I/O limitations compared with native physical systems. When I was working at Intel, our team participated in Microsoft’s Windows Server 2008 Technology Adoption Program. As part of the program, we tested an upgrade of Intel’s entire corporate Active Directory (AD) from Windows Server 2003 to Windows Server 2008 in a special lab at Microsoft’s Enterprise Engineering Center. To do this we used Virtual Server and promoted a virtual AD domain controller into each production domain, copied them to a USB drive, and hand-carried them to the lab. One of our unexpected lab learnings (after spending hours and hours waiting for domain controller promotions to complete as we expanded our little portable Intel) was that Virtual Server had really slow I/O performance.

Fortunately, Hyper-V has a completely revamped I/O model that’s vastly improved over Microsoft’s previous server virtualization product, and it’s gotten better with R2. Before this release, it was a rule of thumb that for maximum performance you should always have virtual hard disks (VHDs) that were fixed in size rather than dynamic disks that would grow as the VM’s space requirements dictated. Tony Voellm, the Hyper-V Performance team lead, has posted a list of performance improvements between Hyper-V’s initial release and its R2 update. One of the more notable improvements is that dynamic disk performance in R2 is now nearly identical to fixed disk performance. The simplified management and space savings you’ll get with dynamic disks will outweigh the tiny difference in performance with a fixed disk configuration. Be sure to have disk defragmentation enabled in both the host and guest, however.

Hyper-V allows you to control how a host’s VMs control the host’s processors (or processor cores, when you have a multicore machine) in three ways. You can have a VM set a reserve on processing resources so it’s guaranteed a certain percent of processing time. You can set a maximum percentage of processing percentage to be sure a VM doesn’t use too much processor resource. Finally, you can set a relative weight on a VM so it’s favored to get more processor resources in times of heavy usage. You should use these controls very sparingly. No matter how you tweak these controls, you only have so many physical processors or cores to go around, and favoring one will always hurt the other VMs on the host. A circumstance where you might want to use processor weighting is where you have an important, transactional line of business VM on a host with less important VMs. However, it’s also reasonable to assume you have this system in a cluster with real time migration, so if it suddenly has to move to another node in the cluster, this favored machine will be throwing its weight (if you’ll pardon the pun) around on the new node and affecting its VMs.

Just like with physical systems, the single best performance boost you can give your VMs is to allocate them plenty of memory. At the same time, physical memory on a host is also its most limited resource. How much memory should you give the host’s parent partition and its other VMs? The simplest advice is to size them as you would size a physical machine, plus a small amount extra for VM overhead (32 MB for the first 1 GB of virtual RAM plus another 8 MB for each additional GB of virtual RAM). For both security and performance reasons, you shouldn’t install other roles or applications on the parent partition. It should have between 512 MB and 1 GB of memory to ensure it has plenty of memory for I/O virtualization, management, and snapshot support.

In the network subsystem, the best way to ensure good performance of a VM within a host is to have Hyper-V Integration Services installed and using the latest synthetic network device drivers. These device drivers take advantage of the VMBus I/O architecture, which is much faster than the legacy emulated device drivers. When you’re looking at new physical network devices for your virtual infrastructure, look for devices that support Virtual Machine Queues (VMQ) and or Chimney Offload. These network performance features are newly supported in Hyper-V R2.

As with the company’s operating systems, Microsoft is trying to make Hyper-V relatively self-tuning. And like its Windows OSes, the best way you can improve Hyper-V’s performance is to give it more resources. In performance tuning, there will always be a bottleneck in one of these computer subsystems because one will always be slightly more constrained than the rest. Once that constraint is eased, the load will shift to what is now the most constrained subsystem. Performance tuning is all about minimizing these bottlenecks in turn until they’re so small your system doesn’t appear to have any constraints at all.

Performance characteristics for VMware VMs are different from physical servers because VMs use shared resources differently. Specifically, you need to be aware of how sharing CPU, memory, disk I/O, and network I/O affects performance. In a physical system, these resources are managed by one instance of the OS, but in an ESXi virtualized environment the hypervisor shares these resources among several VMs. Because the hypervisor manages resource allocation there are performance considerations for the VMs: No single VM can get all of the available host resources; because hardware performance counters are not virtualized, applications cannot use hardware performance counters for performance profiling in the guest OS; and resource availability reporting by the guest OS is inaccurate. In order to optimize your virtual infrastructure for performance, you have to understand the implications of these considerations.

When you think about CPU sharing in ESXi, be aware that a feature called CPU affinity can hurt performance if you use it incorrectly. The CPU affinity feature of VMware ESXi is the mechanism that lets you dedicate CPU cores at the VM level. The result is that you restrict the CPU cores on which a given virtual machine is allowed to execute. When you restrict CPU scheduling and the restricted CPU is not available for scheduling, the VM will accumulate Ready time, which is the time a VM has to wait in a queue in a ready-to-run state before it can be scheduled on a CPU.

Memory overcommit in ESXi hosts gives you greater VM density. But it is effective only when you also enable memory ballooning (i.e., the ability of the hypervisor to shift memory from idle VMs to busy VMs) and transparent page sharing (i.e., the ability of the hypervisor to identify memory pages that are identical across VMs and consolidate them into a single copy that is shared across the VMs). These two memory management mechanisms increase the efficiency of how physical memory is used. Memory overcommit leverages these optimizations and makes it possible to allocate more memory to VMs than there is physical memory in the system.

Failing to take disk I/O needs into account in your system design will affect VM performance. When disk is shared across VMs, they will simultaneously try to access the disk. This means the VMs will be competing for disk I/O, which includes both drive I/O and adaptor I/O.  It is important to identity optimum performance limits of your storage solution and use them as threshold limits.

When you plan for your VM network, you need to build in redundancy and set limits such as average bandwidth, burst rate, and peak rate on outbound VM traffic. Network sharing is implemented by the use of Virtual Switches and Virtual Port Groups. Virtual NICs are connected to Port Groups that are configured on Virtual Switches, which in turn are connected to physical NICs. It is possible to use a single physical network interface to connect a single Virtual Switch and route all network traffic on ESXi host through this interface; however this is far from being ideal as this interface would be a single point of failure.

Planning your P2V or V2V migration without making sure you do certain things—like consider what tools you’ll need, evaluate capacity needs, cover yourself for disaster recovery, and prepare for potential service outages—is like betting on a game of chance. You might be lucky, but odds are that you’ll end up with a mess.

If you don’t have the right tools, you’ll end up with a lot of manual and time-consuming work. For migrating from a physical server or an existing VM to a new VM on an ESXi host, the migration tool of choice in most cases would be VMware Convertor as it supports most of the popular guest OSs. In the edge case scenario, where you might want to directly convert VHD (the Microsoft format) to VMDK (the VMware format), there are third-party and open source tools such as StarWind V2V Converter and VMDK (VMware) to VHD Converter .  Without migration tools, you are reduced to installing the OS and applications manually and restoring data from backups. That is truly a mess.

If you don’t know the peak performance characteristics of the systems you’re migrating, you won’t know how to allocate resources to the VM. The performance characteristics you need to consider include CPU, memory, and network bandwidth.  Less-than-optimum allocation of resources will lead to performance issues, which means you will end up having to migrate the VM yet again to a different host.

If you don’t update your disaster recovery strategy after you migrate, your backup solution might not be aware of the new VMs. As a result, you won’t have any backups to restore from in the event of a system failure.

If you don’t plan for service outages when you’re migrating you’re likely to have very unhappy users, not to mention potential revenue loss.  Scheduling migration at a time that has the least impact on the business and users, along with an effective communication plan to warn of service outages, is enough to mitigate this risk in most cases.  

Planning for migration requires you to think about things that you probably consider already taken care of. But virtualization adds some complications—such as customized hard disk layout, changing HAL type from PIC to APIC, and changing disk controllers from IDE to SCSI—that can trip you up if you’re not planning for them specifically for virtualization. Why not improve your odds of success by making sure you’ve covered all your bet.?

Network considerations in solution architecture are not the same for physical environments and virtual environments. Four important network considerations in a virtualized infrastructure are connected with network strategy. First, you need to have an isolated network for administration to provide guaranteed access to VMs and hosts to prevent network traffic conditions from inhibiting your ability to manage your virtual environment. Second, a separate storage network will help ensure continuous connectivity between VMs, hosts, and storage media. Third, certain management functions are bandwidth intensive and could impact your production network. This means you need an isolated network for these functions. Fourth, fault tolerance depends on uninterrupted connectivity, so you need fault tolerant VMs on a isolated network. Let's look at each of these four considerations.

Your ability to manage virtual hosts and VMs is only as good as your ability to access them. With an isolated network just for management purposes, you are guaranteed access. In addition, you're shielded from other network conditions such as the lag you find in typical production networks.

Storage protocols such as iSCSI and NFS require guaranteed bandwidth and continuous connectivity between VMs, hosts, and storage media. Interruptions and lag due to network congestion can cause data loss. This is the reason for routing storage related network traffic on an isolated network.

In a VMware environment, in situations such as immanent hardware failure, you can use vMotion to mitigate service outages by proactively moving live VMs to different hosts. However, this is a very bandwidth-intensive solution, and if you would mix this traffic with your production network, there is a very strong potential that network delays and failures will affect users and applications. In addition, vMotion network traffic is unencrypted, which is an important security consideration. I recommend you use a dedicated network for this function.

VMware Fault Tolerance (FT) and VMware High Availability (HA) are not the same. FT leverages VMware vLockstep technology to establish and maintain an active secondary virtual machine that physically resides on a different host and performs the same sequence of instructions as the primary VM. Both primary and secondary VMs observe the same inputs; so the secondary VM can take over at any time without data loss or service interruption. Primary and secondary VMs, though on two separate physical hosts, are managed as one. In order for this solution to function successfully the network link between primary and secondary VM has to be guaranteed.  This is the reason for isolating this network on a separate network.

When I'm builging out a VMware ESXi environment, I'm always aware of these considerations and make sure I address them. For example last week, I built out an environment in which each of the servers had six gigabit Ethernet ports. With the four network considerations in mind, I allocated one network port each to an administrative network, storage, management and fault tolerance. The two network ports were teamed together for redundancy and allocated for production network.

You’re putting together your plan to implement server virtualization in your company, and it’s going well. But this great plan isn’t much good if you can’t move data into and out of your virtualized environment efficiently. That’s what networks are for, and a virtual infrastructure has unique requirements. Have you planned for all of those requirements?

Most production server networks in middle-sized and large companies have a single switched network with Gigabit Ethernet (GbE) capabilities, though plenty of shops are still using 100 MB Ethernet. The first network requirement you encounter is that best practices for Hyper-V hosts dictate that the parent partition’s network adapter be on its own network, separate from the network the host’s child partitions use. This is for security reasons as much as for traffic reasons: A network attack directed at one or more of the host’s VMs will be less likely to impact the management of the host.

The next requirement, and it’s a biggie, is that your host servers must be able to handle the aggregate traffic workload of the VMs they host. In other words, if you’ve consolidated twenty physical machines onto a single host, that host must be able to sustain traffic to all twenty machines simultaneously. Though this assumes a worst-case scenario where all the VMs are highly utilized, in network traffic intensive systems, this precaution is not at all unreasonable. Even if you’ve consolidated lightly used servers with modest traffic requirements and could relax that requirement quite a bit, remember that under-resourcing your virtual infrastructure’s network limits its future flexibility and capability.

Perhaps your current network has a dedicated backup subnet as well. That’s great, because you’re certainly going to need one for your virtual infrastructure. Backup is every bit as important as your physical systems, and backing up VMs means you’ll be moving very large files on that network.

Next, consider the provisioning and deprovisioning traffic. System Center Virtual Machine Manager (VMM) has a library that stores software CD and DVD images (ISOs), prepared hard drive images (.VHDs), and even entire VMs. The VMM library is essentially a file server network share, controlled by VMM. Most library operations (such as creating a new VM from a template) involve copying very large files to or from the library to host machines. A fast, perhaps dedicated subnet is critical to quickly provisioning new VMs or moving production VMs to an “on the shelf” status in the library.

Finally, there’s the network within a cluster to consider. A fast network is critical to a cluster’s function, because it controls the I/O speed between the host and its hard disk shared storage. One of the main advantages of using clusters is that it allows you to take advantage of real-time migration capability such as Live Migration or VMotion. A major component of how fast a VM migrates from one cluster host to another is the speed of the cluster’s network. In a cluster that uses the iSCSCI protocol to link its shared storage to its hosts, you really need 10 Gigabit Ethernet (10GbE) to ensure it’s not the slowest component in the I/O chain, and 10GbE adapters and switches are very expensive compared to GbE.

Fundamental physical system performance principles apply to virtual systems too. One of these principles is that there’s always a bottleneck somewhere in the system; performance tuning just moves it around. Therefore, in your virtual infrastructure network design you have to strike a balance between designing for worst-case load and being a bottleneck for the capabilities of the other components. And it can be a significant part of your virtualization infrastructure budget as well.

In his most recent post, Satish talked about how a VM’s isolation from its underlying hardware as yet another advantage over a traditional physical implementation. It’s usually very difficult, and often impossible, to recover a failed physical system from backups to a server with a different hardware configuration. In contrast, you can restore a VM to a different host with minimal trouble. But where are you keeping those backups?

Like Satish, I strongly recommend a standardized hardware spec, limiting the number of server configurations to as few as possible. The capital costs of purchasing the equipment may be higher, but they’re one-time costs amortized over the life of the server. Higher maintenance costs of managing a wide variety of server configurations, however, are recurring costs, and the disruption in daily operations goes beyond even that. In midsize to enterprise companies you’ll have two or more server classes, for example a small web front end server and a large infrastructure (SQL, Exchange) server. The hardware lifecycle will also ensure you always have at least two generations of any given server class, with one generation in main production and another generation either phasing in or phasing out. So even keeping configurations as standardized as possible shows you’ll still have four to keep track of.

An important aspect to your recovery strategy is offsite backups. Onsite backups are great for VM or host failure, but that’s not enough in a data center disaster scenario (aka the “smoking hole in the ground”). Here’s where yet another advantage to virtualization shows up: It’s very easy to move VMs off site. This isn’t such a big deal when you’ve paid for a fully-featured recovery solution because they have offsite capabilities. IF you’ve implemented them; it’s been my experience that off-site storage strategies are often the lowest priority in a recovery solution deployment, and this is a big mistake. But VMs make it easy for midsize and small companies to implement an offsite storage or recovery because unlike traditional backups, the files are self-sufficient. If you can’t afford to purchase expensive recovery software you can use Windows Server Backup or simply export the files from Hyper-V. Then it’s an easy procedure to bring the VM up in an alternate location on alternate hardware. When I was working for Intel, we tested beta versions of Windows Server 2003 and 2008 Active Directory by copying domain controller VMs of our entire production forest to USB drives and loading them into a Microsoft test lab, so offsite VM restoration does scale.

Examine all parts of your backup and recovery strategy with your virtualization hat on. You’ll find that you have opportunities to save money and add flexibility you’ve never had before.

Disaster recovery (DR) can be a major factor in your total cost of virtualization ownership (TCO) considerations. And, if you combine virtualization’s DR capabilities with your management solutions, you can maximize virtualization’s DR benefits and end up with a solid, cost-effective DR  capability.

Have you ever seen an entire datacenter of identical hardware clones? In my experience, that situation seldom occurs. You always have a few machines with different configurations or that come from different manufacturers.  When you’re looking at DR, this common situation can have important implications.

Consider the following scenario: A server in your production environment suffers a hardware failure, and you are tasked with the recovery job. Before you can restore and recover from this failure, you have to:

·         Identify server hardware with a hardware configuration that is identical to the failed unit

·         Load  and patch the OS so that it is identical to that of the failed unit

·         Deploy applications and configure application parameters so that they are identical to the setup on the failed unit

·         Restore data from the backup media

·         Resume production

Accomplishing all this typically takes several hours. If you don’t have automated solutions for this situation, the entire process is also prone to human error.  In addition, you have to maintain an inventory of all the servers of all configurations and manufacturers that you have in your environment. This very costly.

Virtualization encapsulates the OS, applications, and data together in a VM. If one of the production virtual hosts experiences a hardware failure, you can restore the VM on a new virtual host by restoring the files that constitute the VM—and you are back in production.  As the hypervisor abstracts the differences in the hardware of the various virtual hosts, you eliminate the need to maintain a huge hardware inventory.  

Restoring a VM is far simpler, more economical, and takes considerably less time than restoring a physical machine. If fact, virtualization’s DR capabilities are a key driving factor for evaluating its benefits.