Booth

Ahead is again sponsoring a booth in the solutions pavilion.  It will be staffed by some of our executives, as well as members of our engineering staff.

Magazine

Ahead has published a magazine that will distributed as part of the conference materials. It has some of our recent solution briefs, as well as some commentary by engineering on the state of the industry.

Partner/TC Conference

A number of our engineers will be attending the partner conference this year, as well as sessions in the EMC TC conference.  We also have several NDA sessions set up with product management as we research EMC’s products.

Social Media

We will be blogging and tweeting most of the conference, especially the keynote sessions. If you don’t already, you can follow all our updates at www.thinkahead.com/blog, or at our twitter list – @ThinkAheadIT/EMCWorld2012.  We are going to try and cover all of the announcements and add our perspective, as much as time allows for.

Video

We participated in the creation of a video for EMC’s Transforming the Future site, look for it during the partner conference and the general sessions.

In addition to all of this are the sessions, dinners, events, of course.  We all look forward to seeing you there!

Luckily, we were able to design a solution that met their needs, at a pretty reasonable cost.  They already had a VNX array dedicated to the VDI workload, so rather than do something like double the spindle count and switch to RAID6, we decided to look at EMC VPLEX.  The added benefits were just too good to pass up.  They could achieve their uptime requirements, and gain independence from underlying disk subsystems, while also avoiding nasty fork-lift upgrades at the end of maintenance cycles.  Enter: Storage Virtualization.

This design uses a modest single engine (dual director) VPLEX local configuration to present Raid 5 volumes from each VNX as mirrored virtual volumes.  The ESXi hosts use these mirrored volumes for datastores and are none the wiser that under the hood we are alternating reads from each VNX and asynchronously writing to both VNX arrays.  We also have the option of presenting Raid 6 NL-SAS based volumes directly from the VNX arrays in order to save cost where performance isn’t as critical.  I won’t get too far into the weeds regarding IOPS and volume design here, but am happy to answer any questions  about them.

One other added benefit is that between the EFDs for FAST Cache, and the large global cache VPLEX provides, the VMware View replica images (100% reads) are served almost completely from cache.

Check out the diagram and leave your questions and feedback below!

VMware has been concerned about Microsoft since Hyper-V was initially announced.  I think the concern over Microsoft was a key driver in the appointment of Paul Maritz to VMware’s CEO role, as he would fully understand how to combat his former employer.  And you saw from his first day his strategic plan to help VMware flourish: shift the value proposition of the company away from the core hypervisor to higher value areas like cloud computing, orchestration, and application development.  VMware knew that sooner or later Microsoft would get Hyper-V to be a fully functional hypervisor, and if that was all that VMware had in their tool kit, they would suffer massive losses.  So they had to change, all of which is in line with Stephen’s commentary.

At a corporate level, you see the change in full effect, as Scott effectively pointed out.  All of VMware’s corporate messaging is about advanced functionality.  VMworld is awash in messaging about Spring, GemFire and Zimbra.  VMware.com touts heady sounding things like “Application Modernization and Cloud Management”.  In the field, however, I see something different.  I see field level customer engagement, on a client by client basis, still primarily focused around hypervisor licensing.  There are maybe some extensions to that, around VDI, or VC Ops, but the conversations and the projects are usually around the number of VMs per physical host, and how to increase that efficiency.

I think the reason for the lack of field progress in adopting the application and orchestration products can be directly tracked to VMware’s longstanding go to market model, specifically the VMUG approach.  The VMUG and the strength of VMware’s customer advocates has always been, and continues to be, VMware’s biggest asset.  I personally find the VMUG community invaluable.  VMware customers do much of the selling of VMware products in this capacity, and help drive the message and adoption of VMware’s technologies.  This VMUG focus started with the early days of VMware, where the technology was for test/dev use cases, and required only a small investment to get started, like free GSX.  The fundamental problem, however, is that the people who are active in the VMUG community are by and large core technologists.  They are system administrators, architects, and line managers.  They are the people that live and die with VMware, and it’s their lives that are most impacted by the technology.  It is rare that you see an audience of entirely infrastructure directors, application directors, or C-level executives attending one of these events.  It is those very directors and  C-level executives, however, that make the types of decisions that would result in deploying Spring, Zimbra, or cloud technologies.

So on the one hand you have a corporate vision to move up the IT value chain and into the executive decision suite in your installed customers.  On the other hand you have an unmatched user base that is loyal to your technology, but which typically doesn’t impact those application decisions.  What you need to close that gap is a sales force in the field, whether partner based or VMware badged, that can call on both groups simultaneously.  They need to both support the installed data center folks, and attempt to  open up the doors to the other aspects of the business.  They need to do all this while meeting quarterly sales targets.  I think that the over-reliance on the successful VMUG (a.k.a. playing to their strengths), by both VMware and especially by their partners, has stunted the C-Level advancement.  There are some successes in this area (notably the VMware Visioning Team), but it’s not standardized.

On the flip side, lets consider Microsoft’s footprint and approach.  They have long standing relationships with C-level and applications people.  They meet regularly with these decision makers.  Microsoft’s licensing footprint in these organizations runs into the tens of millions of dollars, due to all of the different product lines that are installed.  With that level of spend, Microsoft often has meetings with CFOs as a key supplier.   Microsoft is taking the approach with Hyper-V that VMware took in the early days – use our “free” hypervisor for your test/dev environments, rather than pay the “VMware tax” on those low impact environments.

With Hyper-V 3 however, the game completely changes.  It’s the “later” that VMware was predicting.  Microsoft, from what the early press is showing, will have with this release a mature, fully functional hypervisor.  A hypervisor that can compete effectively with vSphere in the production space, and can effectively operate at a data center scale.  And with Microsoft’s licensing and flexibility, they have a cost advantage, or at least a financial leverage point.  And with access to the financial decision makers, they could trump the traditional VMware champions, and move toward the new technology with less technical input than in the past.  If it’s touted as equivalent technology, and it can save money, financial stakeholders would almost be remiss not to consider the option.

If Microsoft is successful in attacking the core hypervisor revenue, they may be able to critically wound VMware as a company.  That’s my concern, that VMware hasn’t sufficiently diversified their revenue streams, even though they have diversified their products, and that competition in the hypervisor market will negatively impact the company’s finances.  I don’t think VMware would cease to exist, but it’s growth trajectory could be significantly stunted.

I hope this is a case of chicken little, and that I’m proved out wrong.  I love the VMware technology, and their vision, and I have faith in their leadership.  I’m also a big fan of Microsoft technologies, and I see a significant amount of promise with Hyper-V 3.

Disclosure: I am currently long on VMware shares, and work for a VMware partner.

One example that emphasizes this is the VMware View product for virtual end user computing (VEUC) to host desktops. During a vCenter outage, you can still connect to any fully available desktop. However, both types of desktops, Full Clones (sometimes referred to as Thick Clones) and Linked Clones, are affected by the loss of vCenter in various ways. Any desktop type will lose the ability to be powered on, off, or reset by View without vCenter. Linked Clones are especially impacted: any refresh or recompose actions, which are typically automated by the settings of a View pool, are no longer available. This can become a big problem very quickly – imagine a shift change for a call center that is unable to refresh their desktops for the next shift.

While there are many other scenarios that can be impacted by the loss of vCenter, the takeaway here is a need to better protect vCenter from failure. Enabling HA on a vSphere cluster containing vCenter server may no longer be good enough for your use case. Here are a few questions to answer to see how prepared you are for an outage:

1. Have you considered how long it would take you to restore your vCenter server to full working state, including all plug-ins and 3rd party applications?
2. Do you have a runbook or disaster process for getting vCenter back to full working order, with planned testing periods?
3. How often is your vCenter database backed up, and does that time period meet reasonable expectations of lost administrative time?
4. Do you have a repository of all domain or database accounts necessary to re-create your ODBC connections and service logons?

Fortunately, there are many different ways to tackle these problems, including the most robust method: vCenter Heartbeat. The point of Heartbeat is to provide a completely disparate set of data so that nothing is shared, eliminating any single points of failure. A completely unique copy of the database and vCenter server is maintained on a secondary server (or servers, if you separate your vCenter server from the database server) with a heartbeat network. If an outage occurs, the secondary server can assume the primary identity to restore functionality very rapidly. If that isn’t your cup of tea, there are also a variety of backups, mirrors, clusters, or log shipping processes that can be custom designed to offer similar levels of protection for the two tiers.

From June 2011:

Profiling Oracle 11g Performance on vSphere with vscsiStats

One of my larger projects recently has been to virtualize a Tier 1 Java application on vSphere for the State of Wisconsin.  This application previously ran on WebSphere 5 and Oracle 9i, on some older physical boxes.  As part of the upgrade we’re moving to WebSphere 7 and Oracle 11g, all 100% virtualized on vSphere.  It was a big sell for the CIO and department, and so far it has been working great.  Our production go live is the upcoming July 4th weekend.

As part of a bunch of Load Testing, I took a few minutes to grab some storage performance data using vscsiStats.  A full dive into vscsiStats is out of the scope of this post, however I will link some great resources at the end of the article.

First, a bit about how I designed the Oracle environment.  Taking advantage of Oracle ASM was really a big plus, however a lot of people were pretty nervous about the extra layer of management/abstraction (myself included).  For the purposes of the design, however, it ended up working pretty well.  In general, this design was really meant to meet a lot of the best practices regarding Oracle 11g on a vSphere platform.

Here’s a look at how one of the Database VMs was designed.  Notice that there are really two ASM volumes, DATA and RECOVERY, that were designed to meet those obvious database needs.  Both volumes are set to External redundancy to really allow our HP EVA 8400 to do the striping.

Each volume is served by a handful of .vmdk’s on separate dedicated datastores.  These datastores are then mapped to LUNs in a 1:1 fashion.  The disk size, for now, is 250GB, meaning the DATA volume has 4 disks for a total size of 1TB.  In addition, the disks contributing to the DATA volume are on a separate virtual SCSI controller to take advantage of the paravirtualized SCSI driver.

Now for the interesting stuff.  Here is some data gathered by vscsiStats collected over approximately 15 minutes during a load test designed to simulate about 85% of our normal production load.

The first interesting statistic to look at is the average IO size.  Here is the IO Length profile for the DATA disks and RECOVERY disks respectively.  Each member of the group shares a similar profile.

From the graphs we can see the majority of DATA IOs are 8KB.  Conversely, the majority of IOs for RECOVERY are 1KB or 16KB (75%/25% mix).  This tells me to optimize my corresponding SAN disk groups for this IO profile (stripe size).  I believe the EVA has a hard coded 128KB stripe size (which is also the ASM default) so I guess the point is moot.  Unfortunately in my world I don’t have spindles dedicated to this application, but it might be a good justification to management about why we need more disks.

Next, I wanted to take a look at the access profile to confirm what any DBA/SA would expect.  The DATA disk group should be highly random in nature, and could benefit from faster spindles (15k SAS or SSD).  The RECOVERY disk group, meanwhile, is handling logging and should appear highly sequential.  This could be a case for 10k disk or even NL-SATA.  The following graphs confirm.

By and large, we see successive R/W opperations having to travel the maximum positive or negative distance from the previous IO about half the time.  However there is what I would call a “less random” aspect to this as well.

The RECOVERY group (logging) is about as sequential as you can get.

Finally, I wanted to take a look at what effect all of this had on overall latency.  The RECOVERY volume shows sub .5ms latency about 75% of the time, which feels pretty decent to me.

Latency on the DATA volumes is somewhat less than ideal, in my opinion.  About half of the IOs are coming in under .5ms, while the other half are taking longer than 5ms, with up to a quarter taking 15ms.  However with multiple disks serving the volume the burden is eased somewhat.  We’re getting fairly decent average transaction times of about .114s per transaction, so not bad.
EDIT: In retrospect, I was a bit conservative regarding the DATA volume latencies, especially when compared to the previous environment.  If you look at read and write latencies for this disk group individually it was actually quite impressive.

All in all, vscsiStats is an outstanding (free!) tool for profiling your storage workloads.  This is especially great in our case as we justify to the world how virtualizing tier 1 Oracle workloads on vSphere was indeed a great idea!

Please feel free to leave comments, questions, concerns, general theories, etc!

Extra Resources:
http://communities.vmware.com/docs/DOC-10095
http://www.yellow-bricks.com/2009/12/17/vscsistats/

A lot of times there is a misconception of what’s included with Cisco UCS compared to what’s included when purchasing traditional servers. UCS is more about a server architecture and less about just your typical server blade, so you need to factor in more components when comparing it to a traditional servers.

Below is a summary of components that you should consider when comparing the cost of Cisco UCS to traditional servers.

Almost anyone reading this post that has experience with servers and storage over the past 20 years understands the basic concept of the RAID group. RAID, or “Redundant Array of Independent (or Inexpensive) Disks”, is a data protection scheme that was invented in the late 1980s to prevent data loss and maintain availability in a time when hard disk drives were both small and unreliable. Remember when a hard disk drive shipped with a known bad sector map? I’m dating myself…

A great site explaining the basics of RAID along with some history is on Wikipedia at http://en.wikipedia.org/wiki/Redundant_Arrays_of_Independent_Disks

Since the 80s, many technological and manufacturing advances have led to a parabolic increase in the amount of data that can be stored on a single hard disk drive. Drives that were tens of megabytes then are now up to three terabytes (a six-figure multiple increase) and this trend will continue. Today, this trend has brought the industry to a point where the application of RAID protection across groups of physical disks is hitting a wall.

RAID 101

Now for some basics on today’s most common RAID schemes:

RAID1/10

Maintain a redundant copy of data on at least two physical disks to protect against a single disk failure in a two drive pair. The capacity overhead is 50% and the performance overhead is that two disk operations have to be performed whenever the server/host performs a write to maintain the mirror.

RAID5

Use an XOR parity scheme across at least three physical disks to protect against a single disk failure in a group. Using the example of RAID5 with 8 disks, the capacity overhead is 12.5% and the performance overhead is that four disk operations have to be performed whenever the server/host performs a write to maintain parity.

RAID6

Use two XOR parity schemes across at least four physical disks to protect against up to two disk failures in a group. Using the example of RAID6 with 8 disks, the capacity overhead is 25% and the performance overhead is that six (or even nine with some implementations) disk operations have to be performed whenever the server/host performs a write to maintain both parity schemes.

The percentages are with respect to the amount of raw storage, e.g. if you have 10 terabytes raw, you get 5 terabytes usable with RAID1 (or 50% of raw), and so on.

The Rebuild Time vs. Capacity Conundrum

When an individual disk drive fails with any of these RAID implementations, the replacement drive has to be completely rebuilt regardless of the amount of data that has been written to the group, i.e. even if no data has ever been written to a group of 3 terabyte drives by an application, a disk failure requires that all 3 terabytes (“white space” and all) be rebuilt on the replacement drive. With a 3 terabyte drive, this process can take days on a busy group and during this time, the level of protection afforded by the group is reduced. This situation has led to RAID6 being the most common scheme applied on larger/slower drives. RAID5 offers less write overhead, but the risk of losing data due to a second failure in a RAID5 group increases as drives get larger and rebuild times increase.

The Performance vs. Protection Conundrum

While RAID6 is the best choice for protecting data on larger/slower drives, it is the least advantageous from a performance perspective when writing data. This contradiction often leads to performance problems. Take a backup-to-disk use case where a DBA exports data to a filesystem for quick recovery. Choosing large-capacity drives protected with RAID6 is the best economic choice for backup data, but this process can have adverse effects on a storage array under this near 100% write workload. The RAID groups are often incapable of keeping up with the pace of writes, which backs up write cache on the storage array and eventually, server performance suffers.

Note that techniques have been developed to better handle the write overhead of RAID protection that involves two parity schemes. For example, an enterprise storage array can sometimes detect a sequential write workload and write out an entire stripe set (both data and parity) in one operation across the group. Another method is to use non-volatile memory to batch up writes to increase opportunities to write out full stripe sets.

Enter Pools

Almost all enterprise storage systems have introduced the concept of pools, which takes many disks and assembles them into pools for ease of management and wide striping across all disks in the pool for better performance on average. For storage systems that have a lineage that goes back to the 90s or earlier, RAID groups are still used in pools. For example, an 80-drive pool might consist of 10 individual 8-drive RAID5 groups and the pool stripes data evenly across all of them. This implementation method allowed manufacturers to bring their existing RAID group approaches to modern pool-based storage, but there is a much broader data loss risk if a RAID group is compromised. More specifically from my example, if 1 of the 10 individual RAID5 groups in the pool has a dual-disk failure, all data in the entire pool is lost given that each RAID group holds its own unique portion. Overall, a single RAID group could have an impact on potentially tens or hundreds of terabytes of data.

The risk is lessened with smaller/faster hard disk drives (faster rebuild times), but the industry continues to increase the capacity per drive to meet demand while consuming less power and taking up less data center floor space.

The Advantages Of Protection At Logical Levels

Several storage systems developed in the past 10 years employ data protection at logical levels vs. the physical disk level to better address the issues described above. Here are a couple of examples of logical data protection schemes in use today along with the associated benefits:

• A NAS system that protects data at the file level only needs to rebuild the actual files that have been written to the filesystem when a drive fails vs. an entire individual disk drive. Additionally, multiple protection schemes can be applied at the file level with less overhead to provide even more redundancy, e.g. the ability to lose an entire node in a clustered filesystem without data loss or downtime.
• A SAN system that employs data protection on smaller logical units above the physical disks and also supports thin provisioning (allocate on write) only needs to rebuild failed drives based on the data that has actually been written vs. entire physical disks. Protection can also be implemented with less overhead in a manner that allows the loss of an entire shelf of drives without data loss.

Summary

RAID groups will be with us for the foreseeable future. The industry never changes overnight and new ways of addressing the risk will continue to be developed, like simply using multiple pools to avoid having too many eggs in one basket. However, more recent storage designs utilize the concepts introduced by basic RAID protection, but they do so in a way that is better able to handle the coming increases in scale and capacity that will continue for the foreseeable future.

• IT Centralization, or a move to a shared service model for internal or external consumers
• Challenges with optimization of data center processes to support this new scale
• Challenges with reporting, chargeback, and consumption forecasting to be able to run these new models
• Competition with shadow IT or a resistance to change

In each case, the challenges and desires were consistent, even though the customers were in different industries and at different phases in the maturity of their projects.

What each organization is dealing with is a complete transformation of the way the run their IT business.  While each understands the need to move to the new model quickly, the amount of change requires a complete transformation of their processes.  Once the processes are defined, there is a subsequent automation and orchestration of those processes through the intelligent application of technology.

What the organizations really struggle with is how to accelerate that change and the implementation of the new pieces.  Humans, and organizations, can only accept so much change in any given time period.  The problem is exacerbated by the fact that for many of these processes the organizations are asking a new type of questions, or attempting things for the first time.

At Ahead we have developed a methodology for accelerating this very type of transformation.  It’s based on proven consulting methodologies, and combined with some top consulting, process and technical people.  I’ve personally experienced the success of helping organizations significantly compress change periods, with measurable success.  It’s with that mindset that as I hear these challenges and customer needs, that I can offer an effective way forward.

Internally, we always talk about the fact that Ahead was designed as a company for the data center revolution, or what has now been termed in the industry as the seismic shift to cloud computing models.  It’s exciting times indeed to see that need bearing out in the industry, and to be able to assist organizations to realize their visions.

If you find your organization going through the same type of transformation, I’d love to hear your perspective.

First thought is how is VFCache different from similar products currently in the marketplace? Well let’s first look at how it compares to a DAS based cache solution.

DAS-based PCIe Flash technology stores the application directly on the Flash card, so in this architecture it doesn’t provide protection if the card were to fail. With VFCache write-through cache, you have protection against data loss as it sends writes to the back-end storage. With the DAS-based technology you’re also limited to the capacity of the installed card, whereas VFCache provides intelligent caching, so data is promoted based on the current working set. This ensures that the right data is placed on the right storage at the right time.

So how does this compare to storage based PCI-e cache? Well the main differentiator here is that VFCache uses server-side caching which places the data closer to the application I/O stack thus providing reduced latency and increased throughput for applications. This also helps offload storage resources and reduces the storage load as more hosts are using VFCache. This by no means implies that VFCache is better than storage-side caching it’s just different. There are situations where storage based cache is a better fit based on your requirements.

So with great technology also comes some caveats andwith VFcache being a 1.0 solution, there are some cavets that can limit it’s use cases:

• Currently only available on rack mount servers.
• Does not support VMware DRS/HA/SRM as it’s treated as a local resource (Biggest caveat, in my opinion.)
• Only accelerates read operations.
• Not supported on Active/Active storage clusters (Vplex)

In summary, VFcache is an impressive technology as it improves application performance and lowers the back-end storage workload.  I think it has some great use cases with high-end databases, analytics, web applications and etc. I just think use cases for virtualized environments will be limited. The technology has great potential and I look forward to seeing it advance in the coming years.

The Way it Has Always Been

When I bring up the concept of chargeback or a rate card for internal ITaaS consumption, I almost invariably encounter the pushback that “the business” won’t accept chargeback, that it’s never been done there, and that it’s a non-starter of a conversation.  If there is a manufacturer in the room, they will typically default to the “next best thing” approach of saying “of course they won’t, but you should do showback”.  The manufacturers seem to believe that showback is a warm pre-cursor to chargeback, and that by giving the business units reports of what their consumption actually costs, it will pave the way for chargeback in the future.  I think showback is effectively a cop-out, allowing the manufacturer to give lip service to the objection without really pressing the real point.

The Logical Flaw

During these projects, I usually get the chance to interview the stakeholders (IT side) of a project,  and often the internal customers (business units).  What I hear is that the IT folks believe/know they are competing with AWS or another external provider.  From the business unit side, they readily admit they are often using an external service, and paying for it every month.  So if the business is willing to pay for the external service, why is internal chargeback such a taboo?

The Way Forward

The difference is that the business units see value in the services the hosted providers deliver, value that is incremental to what they can get currently from internal IT.  It is true that they are not willing to pay for internal IT services today, because today those internal IT services don’t provide a level of value the business feels is worth paying incrementally for.  This is the business side driver for an ITaaS instance, in terms of responsiveness, availability, and/or performance.  If IT were able to operate and provide the capabilities the outside providers are able to, then internal chargeback would be acceptable, as it is delivering that incremental value.

It’s usually at this point that I’m able to explain to the IT staff that the ITaaS instance they are trying to build is more than just a fully virtualized data center instance.  As such, to truly be ITaaS, it will require the value added orchestration, portal, operational hygene, and yes, chargeback components, to differentiate it from the IT of the past.  If they only build the infrastructure, and don’t change the operations, then a lack of appetite will become a self-fulfilling prophecy, though it need not be.