Deep Dive Performance Analysis – Using Microsoft System Center Operations Manager to Monitor vSphere

Introduction

The best part about the virtual computing is the ease of scale, management, portability and consolidation provided; however, this presents unique challenges for the infrastructure team(s) – How do I monitor this constantly evolving and changing environment? What metrics are the most important? Do I keep using the legacy tools we’re used using? Do I procure something that’s purpose built for the virtual infrastructure (VI)? Like most IT decisions, there’s pro’s and con’s to each approach however, what if we took a bit of a hybrid approach? What if the VI could be monitored and reported upon through within existing toolset?

The Important vSphere Metrics

Suppose you did have a monitoring tool, what should even be looked for? From my experience, the most common pain point and area for concern is disk; otherwise known as the vSphere datastore. Disk is a shared resource and performance problems exist for many different reasons. A common pitfall is when slower, legacy storage is repurposed for the VI because it’s what is on hand.  The second common pitfall is misconfiguration and overprovisioning. However, when monitoring disk focus on the following areas.

Disk

• Kernel Latency
• Queue Length
• Device Read / Write Latency

Compute is made up of CPU and Memory resources that are distributed from the parent hypervisor. Just as disk is a shared resource, performance issues always stem from individual VM resource misconfigurations: Oversized and Undersized VMs, Resource Pools, VM CPU and Memory reservations, Out of Date or Missing VM Tools are a few examples. These misconfigurations place extraneous weight upon the parent hypervisor. ESX host CPU scheduling issues lead to VM CPU % Wait and Host % Ready and can stem from oversized vCPU allocations. Host memory over-commit is made possible due to transparent page sharing, ballooning and swapping. Ballooning is a technique that individual VMs utilize to grant guest physical memory back to their parent hypervisor in situations when the parent is out of physical memory. Through VM Tools a balloon driver (vmmemctl) is installed within the guest operating system. This memory management technique allows the VM operating system to communicate with the parent hypervisor allowing the balloon to ‘inflate’ and allocate memory that is no longer in use back to the parent. Needless to say, ballooning can cause performance degradation if guest memory is low and paging begins. When ballooning is not enough, the dreaded swapping occurs. It is a last ditch memory management technique taken by a host to reclaim memory. This is achieved by swapping guest physical memory out to its individual VM swap file on the datastore. For a complete rundown on the vSphere Memory Management Techniques mentioned above visit the vSphere Whitepaper which goes much deeper.

Compute: Host and VM CPU & Memory

• Average Memory Usage (KB)
• Balloon (KB)
• Swap (KB)
• Consumed
• Active
• Average CPU Usage (MHz)
• CPU Ready Time
• CPU Wait

For a complete listing of all performance metrics collected within Veeam Management Pack and their corresponding definition visit the Veeam MP for VMware Metric Definitions Help Center.

Current Monitoring State

Before jumping ahead to the end state, we’ll start with some of the tools that I’ve seen folks use to ‘monitor’ the vSphere VI.

1. ESXTop
2. vCenter Performance Tab
3. Point Monitoring Solutions
   1. Virtual team
   2. Application teams
   3. DBA team

Each of these tools serve their individual purposes. ESXTop provides a great deal of detailed real-time performance information at and individual ESX(i) host level, making monitoring this ongoing very challenging. Also ESXTop does not provide any level of historical performance reporting. vCenter server was designed as a centralized control and management tool for the vSphere environment, not a monitoring or reporting tool. Application monitoring tools have zero awareness of the underlying hypervisor while on the flip side, virtual environment point solutions have zero insight into the application level leaving them completely blind to what’s running within the OS. Put simply, the problem with each of these current state tools is they don’t see the ‘big’ picture.

Future State: Veeam + System Center Operations Manager

The Microsoft System Center suite delivers a perfect solution that allows users to maintain, update, manage and monitor an environment. One of the components of System Center is Operations Manager ‒ a management and monitoring framework. Operations Manager is extensible through management packs (MPs), which contain the knowledge, rules, monitors, graphical views, reports and more to manage and monitor applications and infrastructure components. Many organizations have made large time investments in creating business processes that wrap around the monitors and alarms that are presented through Operations Manager. Veeam Management Pack for Microsoft System Center ™ fills the environmental void that’s created when legacy physical operating systems are run virtualized by providing app-to-metal visibility.

Allow me to simplify. Prior to virtualization folks ran operating systems and applications on bare metal hardware where the Microsoft Monitoring Agent was installed. This provided monitoring teams with the required visibility to effectively monitor the hardware, operating systems and applications. When we virtualize workloads that single bare metal server now runs many virtualized servers and applications where the monitoring agent is still present within the guest, unfortunately Operations Manager is completely unaware of the underlying hypervisor. Veeam Management Pack allows teams to unify both worlds together and get the best of both worlds! Operations folks can still use the tools they’re used to and benefit from the business processes previously built, while the virtualization administrator gets all of the deep vSphere related performance statistics along with historical reporting, capacity planning and intelligence needed to effectively run the VI.

Summary

Monitoring the vSphere environment should not be an afterthought. Using your existing tools and framework aka. Microsoft System Center Operations Manager to monitor and manage the vSphere Infrastructure is simple and easy with the Veeam Management Pack!