EMC World 2016: Unity the story BEHIND the story.

This is, in my personal opinion, one of the coolest stories at EMC World this year – and let me tell you why. What do you think is harder: inventing something new with a clean sheet of paper, or changing the entire car as it’s barreling around a racetrack?

There’s a joke in the IT industry that “The reason God was able to create the world in only six days was because he didn't have to deal with an installed base.”

I think what Unity represents is an incredible engineering achievement – and an incredible story – a wholesale architectural change, while moving, with a massive installed base.

This post will cover what’s new in Unity, but the thing that the most fascinating is the story behind the story. I’ve been lucky to see it from the inside, and today I’ll share it (from my perspective) with all of you.

EMC VNX has an installed base that is well, well north of 100,000 deployed arrays. It has been the most successful EMC platform to date in terms of the number of people and customers, and VNX brings more net new customers to EMC than anything else we do – even as it became increasingly aged.

The VNX was released on Jan 18^th, 2011. That’s more than 5 years ago. For perspective, that’s when the first MacBook Pro was released, and about a year before Windows Server 2012.

Since then – it’s been a huge success.

Now – even before the VNX was born, we were already grappling with how we would transition the architecture and move completely past it’s CLARiiON and Celerra roots, and their deep architectural underpinnings – we needed to move away from the old, but how?

There are other examples of “change the engine while the car is running” in the storage ecosystem playing out, and it is NOT EASY.  I’m not saying we’re perfect, but this is an interesting example that has been occurring in parallel, somewhat “under people’s noses”.

Unity is that – and it’s a whole new platform – we have swapped every part out while not missing a beat in VNX/VNXe.  

Unity will redefine the mid-range unified market.  Its faster, denser, less expensive, and simpler than all of it’s competition (and we’re willing to put our money where our mouth is with a money-back guarantee).  

VNX was the king in this space.   The King (VNX) is dead (though not EOL!), long live the King (Unity)!

For more gory technical detail, and the fascinating story behind the scenes – read on!

As I’ve noted before – it takes a lot of time to harden a storage stack from inception. A rough rule of thumb is 5 years for a transactional stack, 7 years for a NAS/Object stack. Scale-Out/distributed variations are materially harder. This isn’t a matter of “smarter engineers” – it’s rather that persistence stacks are HARD, and I hate to say it, but there’s an incompressibility of the schedule because in part that time is because they get hardened at customers, in deployments, over time.

When you’re a big company (vs. a startup) this is a little terrifying, because you don’t get the “protection” of growing the installed base slowly. For example, as Isilon and XtremIO have ramped like crazy – they both have gone through bumpy periods before they stabilized.

How do we do it for the most widely deployed enterprise storage stack (VNX) on the planet – while minimizing customer and business risk????

As a business, we embarked on an long journey with important milestones that culminate today.  I want take a moment to give a shout out to Rich Napolitano under whose watch this started, and the hundreds of engineers who have invested blood, sweat and tears over the years – what a journey it’s been. 

This has been a journey to redefine, recreate the world best mid-range external scale-up unified storage platform, and do it without risking customers, partners, and revenue. You can literally see powerpoints of the internal strategy in the bottom of this post from 2011.

The journey can be summarized this way (and it’s fascinating to any people interested in this part of the industry – you have to play the long game):

• Step 1: build a new core platform at the kernel level that would allow us to modularize the code, and support containerized packages, as well as clean kernel-mode insertion of things that need to be very close to the hardware. This project was codenamed “C4”.
• Step 2: find a way to test that out (and start hardening) with customers. Thus was born the VNXe in 2011. You can see this milestone in the journey here. The VNXe has turned out to be a huge success – bringing more new customers to EMC than anything we’ve built before.   There are still things smaller than a VNXe, and Unity, but that’s another story for another day.
• Step 3: At the same time that you do Step 2, completely do a ground-up re-write of the block stack. Start changing the design center in anticipation of pervasive flash, and multi-core CPUs – this was jettisoning the old FLARE code base that VNX inherited from CLARiiON, and replacing it with MCx. Without these changes, legacy very “simple” mapping models would inhibit new data services like compression/dedupe, and would leave the platform in the 1990’s in terms of snapshot behaviors and LUN ownership/failover behavior. This was born the VNX2 in 2013 here. Like VNXe, the VNX2 has been a wild success – billions of dollars of revenue, hundreds of thousands of customers.
• Step 4: completely re-write the NAS stack, which was mature – in both good and bad ways. Good – very robust, and with all the stuff enterprises expect. Bad – 32-bit filesystem (seriously!) that meant small filesystems and filesystem limits, a legacy mechanism for filesystem meta-data mapping that meant that NAS/block functions were very disparate, and failover behavior was too slow. What was needed was a modern 64-bit NAS re-write that mapped into the new block mapping code.

Each of these steps were carefully coordinated, and consciously broken into “digestible parts” so that each part would get hardened, tested – and we would ultimately change the whole car while barreling down the highway.

That journey ends today – and a new journey with the new platform, Unity, begins.

Unity will cover the entire market segment that VNX and VNXe currently serve. Well – not entirely. We’ve realized that at the top-part of the VNX bands (VNX8000), scale-out models will rule the roost – and there XtremIO and VMAX All Flash rule the market (expect them to have configurations that move down to cover the top part of that market).

Unity is squarely focused at the external storage use cases that start small (into the <$10K bands, and with all-flash configurations that start at $18K) – and go up to 500 SSDs/HDDs.

How small is “start small”? Well – with Unity – we are covering the VNX market as a whole with the system architecture that looks like a VNXe. 2U, and everything is in there.

How powerful? In practice, about 3x more performance than the previous generation at every scale point, and not just more powerful, but more dense, more simplicity, less power consumed – not a small leap, but a huge leap. Wow.

And the core codebase isn’t just faster, it’s also better.   New mapping, snapshot underpinnings, and a killer advancement in the QoS behaviors where there much finer granularity, and the ability to provide distinct QoS handling for primary devices versus replicas.

Now – if you look at the picture - you’ll notice there are NO “datamovers” or “file blades”. The convergence is complete.

Speaking of the filesystem – I’ll say this strongly: this ain’t your daddy’s VNX.

Unity is a scale-up system – and is EMC’s scale-up transactional filesystem (scale-out non-transactional is OneFS) we came to the conclusion that the requirements for each were different enough that to try to do both could (would?) result in “flying-boat” syndrome – something that is not particularly good at either goal.

The next-gen filesystem goodies (including the neat VMDK on NFS implementation) will arrive to VMAX customers as well via the embedded NAS (the power of containerized storage stacks :-)

And there’s more to the new filesystem than just new “features”. Filesystem fundamental limits are all up – way up!

Furthermore the – while VNX filesystem resilience has always been good, the hardening that has been done is huge. Fast filesystem start and restart, all metadata is checksummed.

Also – with containerized NAS stacks – we can support full multi-tenant use cases, including segregated IP spaces immedaitely, and even overlapping IP address spaces – historically a gap in VNX relative to some of it’s competition.

Again, a huge leap (you’re seeing the pattern :-)

How exactly do we DO all of this?

Answer: it’s all about software - the Unity Operating Environment. All the years of work on “C4” containerization and abstraction, modularized MCx and unified next-generation filesystem (which doesn’t “layer on” LUNs, but directly intersects to the block mapping layer – so the dataservices like snapshots are completely unified across block/NAS use cases)… Well, it’s come to fruition.

All this runs on a thin, light, hardened version of Suse Linux. No FLARE in sight :-)

And if that all makes our customers happy – this will make them dance in the streets: NO MORE JAVA. The entire UI is HTML5 – simple, works nearly everywhere, and is FAST. In fact, we wanted to make sure that we we’re drinking our own Koolaid, so had MadPow put Unity through its paces for full lifecycle tasks – and compare it with others who are well regarded for their usability/simplicity.

We didn’t just stop at the simplest UI in the business, we also built in a ton of new capabilities to make support better.

We’ve improved on the already great embedded support on VNXe, adding CloudIQ – which is a cloud-based lightweight reporting and management tool that has to be seen to be believed (Modern Datacenter Architectural principles apply here – flash optimized, and cloud enabled).    EMC attendees – make sure to hit Susan Sharpe’s session on this topic!

BTW - if you need more reporting and analytics than “comes in the cloud” via CloudIQ, EMC Storage Analytics (using vRealize Operations) will give you everything you need.

Unity is a family – available in “F” variations that are optimized for the all-flash datacenter, and more traditional hybrids. The space for hybrids is shrinking fast – and we believe the all-flash configurations are the sweet spot and have hit the inversion point for most customers. With 3.2TB SSDs available now, and 15TB SSDs around the corner – this trend is not going to stop.

Also worth noting – the Unity Virtual Appliance is also available – and you can download and start using it TODAY here.

With the Unity VSA- there is a version that follows the new EMC model of “free and frictionless” (go to town, just don’t call for support – it’s community support only), and there’s a commercial version for customers who want support for things like edge deployments that can nicely replicate back to the core datacenter.

And of course (happy to say this as the leader of the Converged Platform Division, VCE) – there is a new Vblock platform sporting all this new bad boy – the Vblock and VxBlock 350.   For customers who want to continue with VNX-based Vblocks, the older Vblock and VxBlock 340 systems remain orderable (sometimes it takes time for customers to pick up new standards).

As you can imagine – Unity means we can make the Vblock/VxBlock 350 smaller, and more cost effect (and improve performance).   The performance optimizations we’ve made in the Vblock/VxBlock system design to adapt for all-flash configuration (which strains fabrics) like the Vblock 540 (XtremeIO) and the Vblock 740 (VMAX All-Flash) fit well in the new Vblock 350.

Unity also expands on the deep ecosystem integration VNX2 and VNXe enjoyed – in particular the VMware and Microsoft ecosystems.

Unity directly connects to vCenter for VM-level awareness and mapping. Unity has updates on the VAAI implementations (make sure you look at updated best practices).

Of course – Unity also enjoys unified VVOL support with Protocol Endpoint (PE) support on block and file:

Beyond what’s in Unity itself, Unity will inherit all the Integrations that VNX enjoys with Virtual Storage Integrator (VSI) for VMware, EMC Storage Integrator (ESI) for Microsoft, AppSync (for app-integrated replication), and EMC Storage Analytics – with these dates as our targets:  AppSync (7/11); VSI (5/31); ESI (now!); and ESA (now!)

We also have listened to customers and learnt from the marketplace that we needed to simplify licensing and the whole economic package. Just like XtremIO and the VMAX All Flash – Unity uses a simple appliance economic model. In other words, everything is included. No “per GB” licensing of software.

Beyond “all included” – how about “no maintenance and support jack-up”?    This dramatically simpler model is de rigeur in the all-flash array market, and since we’re leaning hard into all-flash across the portfolio, and that model is used in XtremIO and VMAX All Flash – it will apply to the all-flash Unity as well:

Holy smokes. This ain’t your daddy’s EMC :-)

That’s a whole lot of “new” here.   In the diagram below – every green arrow represents something fundamentally new in Unity.

And the Unity team isn’t stopping here. 

In Q2 there will be an NDU software update that will add inline compression – just like we are with VMAX All Flash – which will have a effective capacity multiplier of 1.5x to 2x.   A nice little “thank you” – and highlights all the under-the covers core code stack changes.  Our earlier efforts to do compression were a less-than ideal post-process, and with material performance impacts. 

I would argue that between VxRail and Unity, we have perhaps the two simplest products for customers to use, for our partners to embrace… And imagine the power of both VxRail and Unity in the hands of the combined Dell/EMC entity (where integration of the powerful Dell supply chain and server portfolio is a huge force-multipler).

Sidebar: Speaking of VxRail – there’s a lot of internal (and external) debate about the storage market transitions – and what SDS and Hyper-converged Infrastructure Appliances (like VxRail) mean to the “mid-range” unified storage market.

What is a fact (data, not opinion!) is that the external mid-range external storage market currently dwarfs (by a factor of more than 10x!) than the total revenue of all the HCIA players and SDS players put together.

There are also plenty of scenarios where the economics ($/GB, Watts/RU, GB/RU, $/IOps, $/GB) strongly favor external storage models over HCIA and SDS models. It’s a fact. Do the math :-)

I’m sure that over time, we’ll see HCIA and SDS models continue to grow and expand – and cannibalize the external 1storage market, including the mid-range. But – that will play out over years, and the customer needs for simple, scalable, performant, reliable external storage are everywhere.

With all this awesome new Unity goodness, our hundreds of thousands of happy VNX customers may be wondering what it means for them.   VNX isn’t going away – and this doesn’t mark it’s EOL to say nothing of it’s EOSL (if you want to know the EOL/EOSL for any EMC platform it’s right on EMC Online Support under your products).

We know that our larger customers take time to qualify/certify new products, but starting today – if was in the market for a mid-range, unified platform, something with great transactional NAS – I would evaluate Unity, period.  Today marks an incredible new beginning.

To all of the players (traditional and the startups) in this section of the market – there’s a new player on the scene, and Unity is loaded for bear, and competition is fun and good for the marketplace!

More importantly – customers – what do YOU think?