Delivers Over 700% Increase in User Density
As an early proponent of virtualization, Wisconsin-based Aspirus healthcare network has virtualized more than 70% of its Windows infrastructure. However, it ran into a roadblock when attempting to bring its Epic Hyperspace environment on board. Early efforts resulted in unacceptable performance, prompting a return to physical servers for delivering the Epic Health Information System’s user interface to 2,500 concurrent users.
DATA CENTER SPECIALIST AHEAD ADDRESSED THE PROBLEM BY DESIGNING A PRODUCTION-SCALE PROOF OF CONCEPT ON THE CISCO UCS PLATFORM. While other Epic customers who have virtualized Hyperspace have reported achieving user densities of only 35 to 40 per server instance, and Aspirus was handling 125 users per bare metal HP BL465c server, the test supported 1,034 users on a single virtualized Cisco UCS B440 blade configured with only 50% of its capable memory.
The Aspirus IT team expects to scale to 1,400 users per server when the transition to a VMware-based Hyperspace deployment is complete in 2012. That will slash server needs from 20 to 7, saving space as well as $250,000+ in upfront hardware costs. It will also enable other virtualization benefits, such as better CPU/memory utilization and improved load management, and provide reliable user access to information ranging from clinical to scheduling, billing and beyond.
Virtualize Hyperspace or drown in servers
Epic Hyperspace is a client-server application that connects back to a Caché database running on IBM pSeries hardware with AIX. Following Epic best practices, Aspirus delivers the Hyperspace client using Citrix XenApp for application presentation to end users and Citrix Provisioning Server to provision the XenApp servers.
That architecture was adopted when Aspirus implemented Epic HIS for Wausau Hospital in Wausau, WI, in 2004. It functioned well initially, but two difficulties arose over time.
First, the user experience began to deteriorate as the nonprofit health system added applications and expanded to include six hospitals, 40 clinics, multiple ancillary services, and more than 5,500 end users in Wisconsin and upper Michigan. Second, with that growth, Aspirus required 20 physical servers just to host the Hyperspace application for 2,500 concurrent users. Continued expansion would require an even larger footprint that would eventually become unsupportable.
“We knew that virtualizing Hyperspace was the answer to achieve the scalability we needed, and we were moving as much of our infrastructure into a virtualized environment as possible, so in 2006–2007 we began running VMware to provision Citrix Presentation Server and deliver the Hyperspace client,” said Glynn Hollis, Aspirus director of IT. “The user experience was satisfactory at first, but the VM infrastructure crashed and burned when we had a client upgrade, because of the new version’s heavier CPU usage.”
After failing to resolve the issue despite assistance from Epic, Aspirus abandoned VMware for the Hyperspace workload and reluctantly retreated to bare metal. Then Ahead entered the picture.
New path with Cisco UCS and Ahead
At the time, Aspirus was exploring the use of Cisco UCS (Unified Computing System) for a disaster recovery project and was working with Ahead to determine how the Cisco UCS blade servers would reduce their cost per compute resource. Aspirus had already purchased several chassis and had four B200 blades on-site, but the IT team wanted to test the waters before taking the full Cisco UCS plunge. Ahead proposed using Hyperspace as a proof of concept.
“If Cisco UCS failed to demonstrate enough positive difference over our HP bare metal servers, we would not have gone down that path. We needed to be convinced,” said Phil Hammen, Aspirus server analyst.
The goal was to use VMware ESXi 4.1 to virtualize the Hyperspace environment in a test scenario. One of the challenges, however, was that there was no Hyperspace load simulator. The test would have to be run with actual clinical users in live production to determine how the servers would perform under a real-world workload.
Despite initial concerns about potentially kicking users offline or worse, Aspirus decided to move ahead with the proof of concept because of multiple safeguards that mitigated the risk. One was that Aspirus’ Citrix architecture limited the exposure by making it possible to gradually ease users onto VMs. Another was that Ahead’s close connections with Cisco, VMware, and EMC provided high-level resources to consult in the event of a problem.
Accordingly, Ahead designed a series of five tests to safely benchmark the separate components as more users were scaled onto a single compute resource. The building blocks included Cisco UCS, VMware, EMC storage, and Citrix Presentation Server.
1,034 users on one blade and counting
The tests were run over a period of five days in February 2011, beginning with a Cisco UCS B200M blade and moving to a top-of-the-line B440 supplied by Cisco with 128 GB of its supportable 256 GB of RAM for testing purposes. By the time the last test was completed, Aspirus had effectively knocked the Hyperspace virtualization ball out of the park with VMware-based user densities 26 times greater than reported by other Epic customers. The test cycle:
- Pushed the workload to 1,034 users on the test Cisco B440—an eightfold (700%+) density increase over Aspirus’ own physical server load with no performance drop-off
- Pointed the way to supporting up to 1,400 users on a fully loaded B440—an elevenfold (1,000%) improvement over Aspirus’ bare metal HP environment
- Yielded a 50% reduction in user login times from 10-12 seconds to just 5-6
With those results, Aspirus knew that it could successfully virtualize the Hyperspace presentation layer, replace its 20 HP BL465c servers with 7 Cisco UCS B200 blades, and continue to scale its Hyperspace environment without building a Mount Everest of 1U servers.
Equally important, Aspirus gained the ability to leverage VMware features such as vMotion, Storage vMotion, DRS and HA for better CPU and memory utilization, automatic load balancing, live migration of virtual machines from one ESX host or storage device to another, and automatic failover in case of hardware failure in a cluster.
“With Ahead’s knowledge and test design, this project has literally written the book on virtualizing the Hyperspace layer of the Epic application stack. No other Epic customer that we know of has come close to achieving this kind of density and performance, and we ourselves didn’t believe it was possible until we saw it happen,” Hollis said.
The result: another milestone on Aspirus’ virtualization road, exponentially easier management of Hyperspace servers, and fewer end-user complaints about sluggish performance of the Epic interface. That’s an epic accomplishment for five days of testing.
INDUSTRY: Healthcare
LOCATION: Wausau, WI
BUSINESS: Non-profit regional health system with more than 60 locations, over 5,500 employees, and a vast network of affiliated physicians in Wisconsin and Michigan
MISSION: Align regional providers together to provide access to care, coordination of care, and high-value service delivery for optimal clinical outcomes
KEY APPLICATION: Epic Health Information System (HIS)
NEED: Virtualize Epic Hyperspace, the front-end portal that provides the user interface to Epic HIS
CHALLENGES:
- Performance degradation on earlier Hyperspace virtualization effort
- Low user density reported by other Epic users
SERVICES: Full-production proof of concept designed and configured by Ahead
TEST ENVIRONMENT:
- EMC CLARiiON CX4-960
- Brocade DCX-12 4G SAN Fabric
- Cisco Catalyst 6509 10G LAN
- Cisco UCS B200M and B440 Blade Servers
- VMware ESXi 4.1
- Citrix Provisioning Server
RESULTS:
- Increased users per server from 125 to 1,034
- Resolved performance degradation problem
- Reduced Hyperspace server needs from 20 physical to 7 VMs
- Saved $250,000 in upfront equipment investment
- Cut user login time in half
- Gained access to vMotion and other VMware features
- Advanced Aspirus virtualization initiative
www.Aspirus.org