By Liliandecassai (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Impala by Liliandecassai

Impala 1.0 was launched back in July last year, and it’s been supported by AWS EMR since last December so I’ve been meaning to have a quick play and also to compare it with a classic map-reduce approach to see the performance difference. It’s not like I don’t believe the promises – I just wanted to see it for myself.

So I ran up a small cluster on AWS – with an m1.large for the master node and 2 core nodes, also running m1.large. I used the US-West region (Oregon) – which offers the same cheap price points as US-East but is 100% carbon-neutral as well :). This was all running using spot instances in a VPC. For interest, the total AWS cost for 24 normalised instance hours (I actually ran the cluster for just over 3 hours, including one false cluster start!) was $1.05.  Using developer standard units of cost, that’s nearly the price of half a cup of coffee! (or since we’re using Oregon region, a green tea?)

Impala

As I’m lazy, I used the code and datasets from the AWS tutorial – and decided to just use a simple count of records that contained the string “robin” in the email address field of a 13.3m row table as my comparison. Here’s how you define the basic table structure…

create EXTERNAL TABLE customers( id BIGINT, name STRING, date_of_birth TIMESTAMP, gender STRING, state STRING, email STRING, phone STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/customers/';

The output is...

[ip-10-0-0-26.us-west-2.compute.internal:21000] > select count(*) from customers;
Query: select count(*) from customers
+----------+
| count(*) |
+----------+
| 13353953 |
+----------+
Returned 1 row(s) in 1.09s

[ip-10-0-0-26.us-west-2.compute.internal:21000] > select count(*) from customers where customers.email like "%robin%";
Query: select count(*) from customers where customers.email like "%robin%"
+----------+
| count(*) |
+----------+
| 66702    |
+----------+
Returned 1 row(s) in 1.73s

A slight aside - Impala uses run-time code generation to compile down the query down to machine code using LLVM, and this introduces a compilation overhead of circa 150ms, but which more than pays back on the majority of queries.  So this is where some of our 1.73s is going.  More about this here.

Pig comparison

As a glutton for punishment, I decided to use pig rather than the more usual hive for the comparison with Impala. The first thing to say - it was way harder, as the aptly named pig is just a bit more foreign to me than the SQL-like niceness of Impala...so there was some desperate checking of cheatsheets etc to remind me how best to do it...

The basic code for the same source data (already loaded into HDFS) is as follows...

CUST = LOAD 'hdfs://10.0.0.26:9000//data/customers/customers' USING PigStorage('|')
as (id:    chararray,
name:  chararray,
dob:   chararray,
sex:   chararray,
state: chararray,
email: chararray,
phone: chararray);
C2 = FILTER CUST BY REGEX_EXTRACT_ALL(email, '(.*)robin(.*)') IS NOT NULL;
C3 = FOREACH (GROUP C2 ALL) GENERATE COUNT(C2) as cnt;
dump C3;

As you can see the pig approach ran 8 maps. The output is as follows (with all the INFO messages and some other noise removed)...

HadoopVersion PigVersion UserId StartedAt           FinishedAt          Features
2.2.0         0.11.1.1   hadoop 2014-04-10 12:11:13 2014-04-10 12:12:26 GROUP_BY,FILTER

Success!

Input(s):
Successfully read 13353953 records (9 bytes) from: "hdfs://10.0.0.26:9000//data/customers/customers"

Output(s):
Successfully stored 1 records (9 bytes) in: "hdfs://10.0.0.26:9000/tmp/temp1725123561/tmp-1782422819"

(66702)

Conclusion

I was just trying it out, so this is not a fair test in some ways - and I didn't try and do any optimisation of either approach. The Impala approach ran about 40x faster, and this was consistent with repeated runs.

ImpalaPigComparisonGraph

I checked out the CPU, IO etc and there was nothing hitting any limits, and CPU consumption when I was alternately using Impala and pig looked like this - load was even across my two core nodes, and the master had it's feet up most of the time...

CPU CloudWatch metrics

I haven't reported the data here, but I also played with some nasty 3-way joins using Impala and the results were really impressive. Obviously though it's horses-for-courses - MapReduce-based approaches like hive and pig will soldier on when Impala has run out of memory for certain query types, or in the event of a node failure etc. But definitely a great bit of kit to have in the AWS EMR toolbag!

empty pocketFollowing on from my post about Google, AWS and then Azure price cuts the other day, there’s an interesting summary of Rackspace’s position covered on TechCrunch. In summary, the Rackspace CTO John Engates explained that they are continuing on the same track of not matching the recent price drops – which is consistent with his blog from July last year where he said…

We at Rackspace don’t aspire to offer the lowest unit prices. We strive instead to offer the best value…

I suspect a key reason is because they can’t afford to play this game of chicken.

Looking at basic storage as it’s easiest to do a like-for-like comparison, Rackspace’s Cloud Files is 10 cents/GB still, so that’s now 3.33x than the entry price for AWS S3, and 3.8x the entry cost of Google Cloud Storage. Whilst I firmly believe that agility is typically a stronger driver than cost in the enterprise market, that’s such a huge difference that I don’t see how a customer procurement department can ignore it. Rackspace is having to move up the food chain as the base services get commoditised underneath them, i.e. focusing on service management, OpenStack, DevOps etc – get (a bit more) niche or get out. I get the “focus on value” message, but it’s hard to show much differentiating value on relatively commodity services like storage. It looks like this price drop was one price drop too far for Rackspace’s pockets. And then there were 3…

PS As an illustration of the positive impact on our customers, we’ve recently re-priced a customer proposal that was already going through the Smart421 sales machine when these price cuts were announced, and it’s resulted in an immediate 17% overall AWS cost reduction. Nice.

 

Fight in ice hockey 2009It’s been a pretty amazing 48 hours or so in the mega-cloud vendor space. We’ve rather lazily got used to continual price reductions from AWS, but this round of Google vs AWS price reductions are pretty special even given this context.

First Google announced some very big price reductions – it was the storage pricing that really grabbed my attention, at 2.6 cents/GB. But for the majority of workloads the compute costs are the dominant component, and so the 32% reduction in compute costs is probably more significant for many. It’s a minor point, but the Google announcement mentioned “reintroducing Moore’s Law to the cloud“, but Moore’s Law is of course finally running out of steam, e.g. according to Intel it’ll be game over by 2020.

AWS have responded with this, but interestingly seem to be calling time on the race to the bottom, knowing that they have a much more credible enterprise offering than Google I suspect. On S3 they’ve almost matched Google but not quite at 3 cents/GB reducing to 2.75 cents/GB with volume. Perhaps the bit that I’m most excited about is the price reduction of the M3 instance range by a whopping 38% (e.g. an m3.large in US-East is reducing from $0.225/hour to $0.140/hour), given that the M3 range is often our preferred weapon of choice these days. That’s a massive bang for your buck.

The next obvious thing to look out for is what Microsoft do with Azure pricing – the assumption is that they will match AWS as per their previous announcement to “peg” their pricing to AWS. Ouch – imagine being an exec and getting out of bed in the morning to find out that you need to drop your prices by 30-80% across the board!

[ADDED 2nd April - Microsoft have done just that - see announcement on their blog here]

So what conclusions can we draw from all this? Well here are mine:

  1. What’s cheapest today is not necessarily cheapest tomorrow – so optimise costs for the long term, not the short term. OK, if you just want some server or storage capacity for a short time then go with the cheapest I guess, but in reality I’m talking about enterprise workloads and it’s never “a short time” – storage is for life, not just for Christmas :), and the cost of moving between providers might outweigh any benefit. Also, the costs are now so low for some workloads (e.g. if I’m playing around with some feature on AWS) that they are trivial anyway – so convenience and sticking with whatever minimises any usage friction are paramount for me.So rather like when choosing a bank to save your money with, where you might want to go for the savings account with the best long term track record of consistent high interest rates rather than the headline grabbing “bonus” offer – when selecting an IaaS cloud provider it’s their trajectory that matters (and hence their ability to leverage mega-scale). It’s not a great time to be a sub-scale (and sub-scale these days still means freakin’ huge) cloud provider unless you’ve got some specific niche offering…
  2. In general, we don’t recommend buying AWS Reserved Instances (RIs) for longer than a 1 year term. The 3 year term often makes more financial sense at that moment in time, but in reality the prices are dropping quicker in a year than the additional saving. This makes sense really, as AWS virtually created the IaaS market only 8 years ago, so a 3 year commitment is still a lifetime in this market.In fact, now is a great time to buy 1 year AWS RIs as it’ll be a few months (you’d have thought!) until the next round of potential price drops – maybe timed to coincide with the next AWS Re:Invent conference in Las Vegas in November – so you’ll get the maximum saving. An exception to my point here is that sometimes 3 year RIs are useful for projects where the TCO needs to be completely fixed and predictable – i.e. cost predictability for a business case is the primary requirement.
  3. A mild concern about where all this is heading – in my view there’s enough competition in the market at present for it to be healthy (i.e. the consumer is winning at the moment), but there is a risk that all but the most massive cloud service providers are squeezed out and the resulting oligopoly means that prices start to creep up again. You could argue that Microsoft’s price pegging announcement is an early sign of an oligopoly forming – reminiscent of the behaviour of the supermarket sector in the UK (where 4 retailers have 76.5% of the market). We’re a few years away from this risk so I don’t think this should infuence enterprise’s investment and vendor selection decisions today.

We’re loving it – what a great time to be migrating customers’ IT workloads to a cheaper, more agile platform where the price is only going down!

CloudExpo2013 Our LocationWhat does last week’s Cloud Expo Europe tell us about the maturity of the market for cloud services in the UK? As an Amazon Web Services Premier Consulting Partner, Smart421 had a stand in the Amazon Web Services Village which gave us a great opportunity to have numerous customer conversations. Wayne Stallwood, one of the AWS Architects from our internal AWS Practice, supported our sales and marketing staff on the stand, and we compared notes afterwards to draw out the key themes.

First of all, one immediate observation was that people were more openly talking about hosting production/live systems in the cloud, i.e. not just the usual dev, DR and startups. We’ve been at this cloud game for about 4 years now and so it is far from new for us (although as a side note, it was interesting to hear Mark Russinovich from Microsoft Azure saying “the cloud is new” this week) and we started to see this shift at least a year ago if not longer. Some of the presentations at Cloud Expo Europe reflected that, for example with a talk about DDoS hardening etc – very much about live systems. There were lots of questions about performance stability, resources, scalability, reliability etc – again more enterprise-level considerations.

Smart421 stand

Smart421 stand

Balancing this though, it was somewhat alarming that some of the people coming to the stand still wanted to talk about the buzzwords without really knowing what they are…so we had a few openers where it was “so this “big data” thing…what does that do for me?” and if you looked at the name badges it was established enterprise people asking the questions. This tells us that there’s still a huge lump of “educational debt” to overcome in the enterprise space.

I had time to attend a couple of presentations but they were pretty awful – dull vendor pitches. You need to choose carefully at these events as the attendees typically don’t pay to attend, and so the bulk of the funding for the event has to be sourced from vendors, and hence they all get to present. There is always some great content though, you just need to be selective and accept you’ll get a few duds.

Instead, I devoted the bulk of my time to understanding Red Hat‘s direction of travel, especially in relation to OpenStack (as I’m fascinated by the cooperation and competition in this area, e.g. from Mirantis , and I’m interested to see how the delivery of private clouds plays out as enterprises use it as a not always sensible stepping stone to the inevitable destination of public cloud) – although inexplicably they were squirreled away on an upper floor and poorly represented in the online show mobile app and so pretty hard for people to find.  I also took some time to catch up with AWS colleagues old and new – including AWS Technical Evangelist Ian Massingham.

The Cloud Expo Europe event itself was co-located with Data Centre World (just over half the floor space) and Big Data Expo Europe (really just a thinly populated stream within the Cloud Expo event), and it was a bit odd to be wandering around the show floor and then stumble into the “dark side” with vendors trying to pitch cooling, racking and UPS power systems to me. I don’t want to build a data centre, ok, AWS has already taken care of that for me :).

The pure cloud content felt smaller to me that previous years, and so as a final thought – I wonder if this reflects not so much that the cloud market is going off the boil but more the opposite – that it’s now mainstream enough that it’s harder to raise so much interest for events that are riding the latest hype?

flock of migrating canada geese birds flying at sunsetIn yesterday’s blog post I summarised the cloud services broker role with some definiti0ns, and concluded that I do indeed appear to work for one of them – and I suspect many vendors at this week’s Cloud Expo Europe (#cee14) might lead on this from a marketing point of view.

We’re delivering service intermediation/customisation and service aggregation/integration, but one thing we are not really doing (or seeing any demand for) at the moment is dynamic or semi-dynamic workload migration. i.e. it’s not just dev & test any more, these days we are migrating complex production environments customer after customer onto AWS. But we are not providing customers with the means to dynamically move or spread those IT workloads across different cloud providers. It’s certainly something we could do from a technology perspective, and most of our deployments have some hybrid aspect to them.

cee_logo_2014_nowebsiteThe ability to migrate IT workloads dynamically (i.e. at run-time, not at deployment time) is something I sometimes see as a capability under the “cloud broker” banner, but in my view it really just doesn’t make sense – at least not at the moment.

The rate of innovation in the IaaS/PaaS/DaaS market is such that most of the other vendors are playing catch-up with AWS, as AWS continue to differentiate themselves from the following pack. This shows no sign of slowing down over the next couple of years – so the only way a migrated workload is going to work across multiple cloud vendors is if it only relies on the lowest common denominator functionality across the vendors, which is typically basic storage, virtualised compute and connectivity.  Or you have to architect your solution to take into account deployment differences across the cloud providers you intend to use – and be able to effectively monitor and support both – twice the work and complexity, and not something you really want to debug.  Did your load balancing just stop working as you expected – it worked last week…mmm…I wonder if our load balancing configuration behaves exactly the same across all our cloud service providers? And so on…

Even storage – the most commoditised of the building blocks of IaaS (you would have thought) contains some interesting differentiation – not on price any more as Google/Microsoft/AWS are effectively price-matching these days, but on features like access control, archiving to cheaper storage, automated data life-cycle policies etc.

The bottom line is that if you are going to architect your applications so they can run on any cloud service provider, then you can’t easily use any of the good bits and hence your value in migrating to a cloud solution is diminished.  Not ruined, just reduced.

There are now a bunch of brokerage tools out there from vendors that claim to give you this workload migration capability, but what I’ve seen so far is disappointing, e.g. one recent new tool I looked at required a custom template to be created for each cloud provider – so whilst the end user might get a menu choice of “deploy to AWS” or “deploy to Azure” – under the covers you still need a bunch of experts in each cloud service provider’s technology and those experts need to keep themselves abreast of new functionality continually.  You can create an impression of homogeneity, but it’s just a veneer.

In our experience, even in very large enterprise estates (e.g. where we’ve deployed and managed up to 750 AWS instances made up of numerous discrete environments), whilst the IT workloads might be relatively consistent in nature (e.g. there might be a corporate standard for this OS and that application server etc), there is always sufficient variance in each project’s requirements that a completely cookie-cutter approach to environment deployment and self-service just does not work.  Each project needs slightly different software, or software versions, or server specifications, or connectivity requirements etc etc – and the list goes on.  And if they didn’t – well – you’d hope the projects would be merged into a single project if they were so similar in their needs, right?

So – given that it’s taken the IT industry this long to get to the point that “as a service” is really possible, for the next couple of years at least let’s focus less on hiding away the good bits of cloud and the really useful differentiating features of the leading cloud providers, and focus more on actually exploiting them please!

PS If you happen to be attending Cloud Expo catch me on the Smart421 stand or let me know you are there via @SmartCTO

CloudBrokerBadgeThe first Cloud Expo event I attended 2 years ago was striking for the myriad of traditional hosting companies who were cloud-washing their offerings (usually quite blatantly and badly I felt).  Last year what struck me was the myriad of new small vendors selling niche cloud-related product offerings – data transfer optimisation, security products, management products, etc. I wonder what the theme will be this year? It’ll be interesting to see how many vendors are wearing the “I’m a cloud brokerage” badge at this week’s Cloud Expo.

Whilst I was at AWS’s Re:invent conference last November, one of the guest speakers at the partner day was Tiffani Bova, Distinguished Analyst from Gartner. Part of her presentation covered the topic of cloud brokerage, something Gartner have been talking about for quite a while, and something Tiffani and I had some debate about afterwards.

I must admit, it took me a while to wrap my head around the concept of cloud brokerage, partially as the pushing of the term was coming more from the analyst community than the rest of the cloud industry. Williams Fellows from 451 Research refers to this as “…a ‘cloudemic’ of firms now stamping ‘cloud broker’ across their service portfolios”. Tiffani’s view was that 90%+ of the AWS partners in the room (including Smart421) were brokers. It’s such a broad definition, e.g. Gartner’s definition is

Cloud services brokerage (CSB) is an IT role and business model in which a company or other entity adds value to one or more (public or private) cloud services on behalf of one or more consumers of that service via three primary roles including aggregation, integration and customization brokerage.

The great thing about definitions is that you can never have enough :). Way back in mid 2011 NIST published the following definition of a Cloud Broker…

NIST CloudBrokerageDefinition

The default view taken in society in that anyone with the title “agent” (aka broker) is looked down upon – e.g. estate agent, recruitment agent etc :). But by this definition I guess we’re all brokers in one way or another, even if it’s just combining toast and scrambled eggs to make the kid’s breakfast in the morning (aggregation).

Looking at what Smart421 delivers for our customers – we integrate different cloud and non-cloud services, we design and implement complex cloud environments and we add a 24×7 ITIL-based service management capability on top including ongoing capacity and cost optimisation. We also add value by handling access management, enhanced security, managing billing complexities and bringing new market innovations to our customers (as cloud service providers like AWS are still innovating and releasing functionality at an amazing rate, too fast for customers to keep up generally).  I guess that means I get to wear the badge too!

In a blog post tomorrow I’ll talk some more about one of the oft-touted cloud brokerage promises – that of dynamically migrating IT workloads across cloud providers (closely related to the dreaded phrase “bursting into the cloud”), and of making deployment-time decisions about which cloud service provider to use…and why I don’t believe in it.

PS If you happen to be attending Cloud Expo Europe this week, catch me on the Smart421 stand or let me know you are there via @SmartCTO

Back in September last year I was one of the presenters at an event run by Qualitest where the focus was on big data – the possibilities, implications, testing ramifications etc.

I focused on how to make it real in your organisation and went through a Smart421 case study of the discovery and productionisation phases of a big data initiative, and the run cost implications when using an AWS Elastic MapReduce-based Hadoop infrastructure to run pig queries.  I’ve been meaning to share the YouTube content but Christmas got in the way :), so here it is…

As part of this event I gave a short 6 minute interview on the subject and my perspective on it…

 

…and here’s the presentation itself…

WhereHasAzureCDNGoneI stumbled across the fact today that Microsoft have quietly withdrawn their Content Delivery Network (CDN) offering from Azure for new customers. Several Azure users report a response from Windows Azure Technical support along these lines…

We’re in the process of building out our next generation Windows Azure Content Delivery Network (CDN) architecture, and during this time we are no longer accepting new CDN customers.

We highly encourage you to wait until we’re ready building our next generation Azure CDN service.

That’s fine – because customers who were already using it have still got it, right? Well no, upon reflection, this feels like a bigger deal than I first thought. In the earlier days of Azure the CDN was a top table feature and selling point, and then with little ceremony it quietly disappears. So little ceremony in fact that forum posters were asking where it’s gone, assuming it was a portal bug. For any cloud provider to remove a key part of their cloud service catalogue whilst they create a better one – it’s just not really on. It also makes me wonder how many people are really using it, as surely there would be more fuss in the blogosphere if they were. To be fair to Microsoft, I get the impression that if I asked for access to this functionality to be enabled for my account I might well get it, but it does raise the question about how enterprise-strength was the CDN service if they felt they had to replace it.

Come on Microsoft – this is not really good enough in the cloud era. As cloud consumers we’ve got to believe that the PaaS services that we are engineering into our solutions are going to be there tomorrow (or at least with a smooth transition capability, on my timetable), or else we won’t bother…

phoenixThe other day I was hearing all about the Phoenix open source project, which has a great strap line of “we put the SQL back in NoSQL”. It’s a SQL skin over HBase that is provided as a JDBC driver, and it’s come out of Salesforce.com, and has been proposed as an Apache incubator project with the vote started yesterday (Thurs 5th December 2013).

What I find ironic about the whole SQL/NoSQL thing is how there is a huge amount of energy being put into “SQL-ising” NoSQL datastores. Obviously hive does it, Impala from Cloudera etc, and now Phoenix on top of HBase. Whilst being really impressive – and I mean that – Phoenix currently has some limitations that just bring home the maturity of the SQL/relational database world, such as the need to define your tables in the right order in your joins to optimise performance etc – features that SQL query optimisers have laughed in the face of for years.

One really nice feature of Phoenix is it’s support for secondary indexes, where under the covers it creates and maintains a separate HBase table but transparently uses it to prevent table scans when it can – something HBase developers have been laboriously hand-cranking for a while.

Also it provides query plans so you can understand what’s going on. In the relational world the query optimisers are so good these days that SQL developers can often be pretty slap dash in writing queries and still get good performance characteristics, at least up to a certain level of scale anyway – you are abstracted away from a lot of the underlying complexity, so can be more productive. Of course there is no substitute for understanding what is really going on under the hood, but in the “SQL on NoSQL” world, you really do need to understand the gory tuning nuts and bolts of the underlying NoSQL datastore or else you’re going to be in trouble.

The reasons behind the origins of Phoenix are compelling – Salesforce.com needed to store potentially millions of data items across many thousands of customers, and so they adopted HBase to deal with that scale. It’s fundamentally very batch in nature and they needed to support low latency web applications per customer. But the key driver for SQL-like interfaces that you hear repeated across all these NoSQL datastores is that well…everyone just knows SQL.

It is the lingua franca of data queries, and for most use cases, broad adoption by your developer community (even inside a very tech-savvy company like Facebook) is worth a heck of a lot more than that last 1% of NoSQL tuning that you might be able to squeeze out using a guru and the HBase API. SQL has proved to be very flexible across a wide number of data models – although the NoSQL community’s use of it has introduced lots of extensions, it’s not like the relational database vendors didn’t do that is it?

Long live SQL!

Vegasvenise1 Now that the razzmatazz of AWS re:Invent is over, I’ve taken to some time to step back from the specific new announcements and consider my higher-level conclusions, and what they tell us about the state of the cloud computing market in general.

As for the event itself, the whole “feel” was very start-up rather than enterprise focused, unlike the AWS Enterprise event Smart421 sponsored back in September. The main thing that struck me at the UK event was that wow – this is getting serious. I’ve been “banging on” for a while about the time now being right for enterprise adoption and the enterprise market reaching a tipping point – but the ability for AWS to draw a 500+ crowd from the UK enterprise market says it all, and also the fact that they felt confident enough to do so – even though there has already been a London-based AWS event earlier in the year. AWS subtly adjusted the tone and message for this audience and got it spot on I think – one of their challenges at other events like re:invent is generally that the audience is a mix of hardcore developers/Dev-op types, architects and enterprise attendees. This enterprise space is where Smart421 is used to operating and so it felt…well…just a bit more like home for us.

Hence I don’t interpret the start-up focus at re:Invent as a lack of credibility in the enterprise space really – it’s more that the new cloud-based startups are where the action and excitement is, and the innovation and crucially the adoption of innovation. I only attended one specifically “old school” vendor presentation at re:invent and compared with the other AWS-led sessions I attended – it was so D…U…L…L… that I just turned off. They demonstrated capability by telling – not by showing or doing.

As you’d expect, there was an army of AWS staff on site, but without exception I found them to be very well informed (there was no “give me your card as I’ll get back to you”…), very good, and uniformly excited about what they were doing.

In an least two presentations I attended there was reference to the flywheel effect whereby more and more momentum is created via a virtuous circle of innovation and customer adoption. Whilst I knew this before I went, I found it very striking that AWS’s success is founded on not out-doing the competition (at least not initially), but on out-innovating them. The crucial thing they have solved is how to innovate – sometimes in leaps, but usually in relatively small increments – but very fast, predictably and mercilessly. This creates some difficult tensions with their partners as mentioned in my previous post – ISV partners are especially at risk of being made redundant, but consulting partners like Smart421 have challenges too, e.g. every time RDS becomes more capable (multi-AZ etc) then it’s more likely to be used in one of our deployments, and needs less professional services to design, implement and support than just a database deployed on a virtual machine.

I can see how competitor X could maybe beat AWS today, e.g. maybe VMware can use their scale to defend their virtual desktop market against Amazon WorkSpaces – but who would take a bet against AWS not incrementally improving their offering faster than VMware? I wouldn’t. Innovation pace will always beat an existing market advantage eventually – it’s inevitable.

So does that mean AWS are unstoppable? Of course not – history suggests that they too will get blown away by another market disrupter in the future. Whilst we know this should be the case over a 100 year timeframe, at the here and now it seems impossible. But there are challenges – one challenge for AWS going forward was hinted at by Senior Vice President Andy Jassy – the person who has taken AWS from inception to where it is today – at the UK Enterprise summit. He came up with the quote of the day…

there’s no compression algorithm for experience

…which for me nicely sums up exactly how I feel about the services we offer to customers today. There’s an underlying tension in AWS’s market positioning in that on one hand they say that anyone can get started on AWS, just sign up and start your first instance etc – and it’s completely true – you can be up and running in minutes. On the other hand, enterprise customers know (or are starting to appreciate at least) that the real long term challenges are about managing change – change of architecture practices in their organisations, change of service management processes to consider capacity management differently, change in finance and procurement teams to get their heads around variable pricing etc. Also, once they get past the skunkworks stage, they quickly realise that this stuff is complicated. To get this point across, here’s all the separate Vision stencil icons representing various AWS services and features that we exploit to create AWS designs for our customers…and this was before all the new announcements at re:invent…

AWS Visio Stencils at Sept 2013

…and this set will clearly continue to grow further as AWS’s pace of innovation is simply exceptional. It’s a toolbox for our architects to pick from, so obviously we don’t use all the AWS services for every customer deployment…but you get the point I’m sure. Being intimately familiar with this ever-increasing set of capabilities and how best to exploit them is a full time occupation for our internal AWS Practice – and so I don’t realistically expect most enterprises to have the desire or ability to do so.

AWS will need to take care to balance their rate of innovation with the ability of the bulk of their customers to understand and adopt what they are offering. But that’s a challenge I’m sure their competitors would gladly take!

Share this blog by using the social buttons below or via the short URL http://bit.ly/1aNVxCh

Please Rate and Like this post.  Our readers want to know what YOU think, so please leave a Comment.

Follow

Get every new post delivered to your Inbox.

Join 1,122 other followers