By Liliandecassai (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Impala by Liliandecassai

Impala 1.0 was launched back in July last year, and it’s been supported by AWS EMR since last December so I’ve been meaning to have a quick play and also to compare it with a classic map-reduce approach to see the performance difference. It’s not like I don’t believe the promises – I just wanted to see it for myself.

So I ran up a small cluster on AWS – with an m1.large for the master node and 2 core nodes, also running m1.large. I used the US-West region (Oregon) – which offers the same cheap price points as US-East but is 100% carbon-neutral as well :). This was all running using spot instances in a VPC. For interest, the total AWS cost for 24 normalised instance hours (I actually ran the cluster for just over 3 hours, including one false cluster start!) was $1.05.  Using developer standard units of cost, that’s nearly the price of half a cup of coffee! (or since we’re using Oregon region, a green tea?)

Impala

As I’m lazy, I used the code and datasets from the AWS tutorial – and decided to just use a simple count of records that contained the string “robin” in the email address field of a 13.3m row table as my comparison. Here’s how you define the basic table structure…

create EXTERNAL TABLE customers( id BIGINT, name STRING, date_of_birth TIMESTAMP, gender STRING, state STRING, email STRING, phone STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/customers/';

The output is…

[ip-10-0-0-26.us-west-2.compute.internal:21000] > select count(*) from customers;
Query: select count(*) from customers
+----------+
| count(*) |
+----------+
| 13353953 |
+----------+
Returned 1 row(s) in 1.09s

[ip-10-0-0-26.us-west-2.compute.internal:21000] > select count(*) from customers where customers.email like "%robin%";
Query: select count(*) from customers where customers.email like "%robin%"
+----------+
| count(*) |
+----------+
| 66702    |
+----------+
Returned 1 row(s) in 1.73s

A slight aside – Impala uses run-time code generation to compile down the query down to machine code using LLVM, and this introduces a compilation overhead of circa 150ms, but which more than pays back on the majority of queries.  So this is where some of our 1.73s is going.  More about this here.

Pig comparison

As a glutton for punishment, I decided to use pig rather than the more usual hive for the comparison with Impala. The first thing to say – it was way harder, as the aptly named pig is just a bit more foreign to me than the SQL-like niceness of Impala…so there was some desperate checking of cheatsheets etc to remind me how best to do it…

The basic code for the same source data (already loaded into HDFS) is as follows…

CUST = LOAD 'hdfs://10.0.0.26:9000//data/customers/customers' USING PigStorage('|')
as (id:    chararray,
name:  chararray,
dob:   chararray,
sex:   chararray,
state: chararray,
email: chararray,
phone: chararray);
C2 = FILTER CUST BY REGEX_EXTRACT_ALL(email, '(.*)robin(.*)') IS NOT NULL;
C3 = FOREACH (GROUP C2 ALL) GENERATE COUNT(C2) as cnt;
dump C3;

As you can see the pig approach ran 8 maps. The output is as follows (with all the INFO messages and some other noise removed)…

HadoopVersion PigVersion UserId StartedAt           FinishedAt          Features
2.2.0         0.11.1.1   hadoop 2014-04-10 12:11:13 2014-04-10 12:12:26 GROUP_BY,FILTER

Success!

Input(s):
Successfully read 13353953 records (9 bytes) from: "hdfs://10.0.0.26:9000//data/customers/customers"

Output(s):
Successfully stored 1 records (9 bytes) in: "hdfs://10.0.0.26:9000/tmp/temp1725123561/tmp-1782422819"

(66702)

Conclusion

I was just trying it out, so this is not a fair test in some ways – and I didn’t try and do any optimisation of either approach. The Impala approach ran about 40x faster, and this was consistent with repeated runs.

ImpalaPigComparisonGraph

I checked out the CPU, IO etc and there was nothing hitting any limits, and CPU consumption when I was alternately using Impala and pig looked like this – load was even across my two core nodes, and the master had it’s feet up most of the time…

CPU CloudWatch metrics

I haven’t reported the data here, but I also played with some nasty 3-way joins using Impala and the results were really impressive. Obviously though it’s horses-for-courses – MapReduce-based approaches like hive and pig will soldier on when Impala has run out of memory for certain query types, or in the event of a node failure etc. But definitely a great bit of kit to have in the AWS EMR toolbag!

empty pocketFollowing on from my post about Google, AWS and then Azure price cuts the other day, there’s an interesting summary of Rackspace’s position covered on TechCrunch. In summary, the Rackspace CTO John Engates explained that they are continuing on the same track of not matching the recent price drops – which is consistent with his blog from July last year where he said…

We at Rackspace don’t aspire to offer the lowest unit prices. We strive instead to offer the best value…

I suspect a key reason is because they can’t afford to play this game of chicken.

Looking at basic storage as it’s easiest to do a like-for-like comparison, Rackspace’s Cloud Files is 10 cents/GB still, so that’s now 3.33x than the entry price for AWS S3, and 3.8x the entry cost of Google Cloud Storage. Whilst I firmly believe that agility is typically a stronger driver than cost in the enterprise market, that’s such a huge difference that I don’t see how a customer procurement department can ignore it. Rackspace is having to move up the food chain as the base services get commoditised underneath them, i.e. focusing on service management, OpenStack, DevOps etc – get (a bit more) niche or get out. I get the “focus on value” message, but it’s hard to show much differentiating value on relatively commodity services like storage. It looks like this price drop was one price drop too far for Rackspace’s pockets. And then there were 3…

PS As an illustration of the positive impact on our customers, we’ve recently re-priced a customer proposal that was already going through the Smart421 sales machine when these price cuts were announced, and it’s resulted in an immediate 17% overall AWS cost reduction. Nice.

 

Internet Explorer running through workspaces on an iPad.

Unusually, I’m writing this blog post in a browser. Specifically, I’m writing it in Microsoft Internet Explorer, on Windows. Not particularly odd you might think, except that I’m not sat in front of my laptop. I’m using my iPad.

It’s ok, you haven’t gone mad. Hell might have already frozen over when Microsoft released Office for iPad last week, but rest assured it hasn’t happened twice: MS have not released Windows iPad Edition.

Instead, I’m trying out Amazon’s new Workspaces product. This product went GA last week, after a private beta announcement which we covered back in November.

Workspaces is a virtual desktop product that allows you to run managed Windows 7 desktops in AWS with very little effort. Signing up takes minutes, and you can provision either a mostly vanilla workspace with just a few basic utilities installed, or a ‘Plus’ workspace which adds Microsoft Office Pro, and Trend Micro AV. In either case, licences for the installed software are included in the price, making it a great way to stand up a desktop machine with all the essentials fully licenced in no time.

There are two performance teirs: ‘Standard’, with 1 vCPU and 3.75GB of ram (which sounds suspiciously similar to an m3.medium instance), or ‘Performance’, which packs 2 vCPUs and 7.5GB of RAM (m3.large, anyone?). As is commonly considered best practice, each machine has a couple of disk volumes attached: one that holds operating system and applications (C:), and one for holding a user’s data (D:). Data on the user’s D: is automatically backed up every 12 hours.

Depending on the bundle you chose, prices range from $35 to $75 per month.

You access your workspace using an amazon provided client application that runs on Windows, Mac, iPad, Kindle or Android tablet.

So, that’s the basics covered. How is it to use? Honestly, from the UK it’s currently a little painful. This is to be expected as workspaces is currently only available in the US, so every pixel of my display is being shot across the Atlantic before I get to see it. I’m seeing latencies of just over 200ms, and Amazon recommend a sub 100ms latency for good user experience. I can confirm that both iPad and Mac clients work well enough (spot the Apple fanboy), although in common with any iPad based remote desktop product, the touch-your-screen-to-point-your-mouse impedance mismatch is disorientating at times. Swapping between devices seems to work much as you’d expect. If you’re logged on from your iPad, and then sign in from a desktop, your session transfers seamlessly to the desktop.

From an infrastructure/desktop manager’s perspective, it’s early days at the moment I think. AD integration is possible, allowing users to log in with their normal credentials, as well as allowing them access to local printers and (I assume) file shares. While deploying your own software is certainly possible, you’re pretty much on your own there: There is no concept of an AMI here, nor is there any support for packaging and deploying applications within the service itself. This in itself is probably not a disaster in some senses, since most enterprises have their own deployment tools, but the lack of custom AMI capability makes boot strapping a workspace into the deployment tool harder than it would otherwise be.

What about use cases? We can already see a couple of things we do for customers where workspaces could replace or supplement what we currently provide:

  • Cloud DR solutions (for an example see our Haven Power case study). As things stand, the key issue preventing us from doing this is the fact that you pay for workspaces per month, regardless of how much usage of the workspace you make. Unusually for AWS, there isn’t an API allowing you to automatically provision/deprovision workspaces, making it hard to optimise the cost here.
  • Remote desktops for 3rd Party users. We deployed a Windows Terminal Services farm in AWS for another of our customers, who use it to allow third parties to work on their applications. Both the applications and terminal services farm are managed by us in AWS, and are accessed globally. in theory it would be relatively straightforward to replace the terminal services farm with Workspaces, although we’d have to be confident that the performance is adequate.

Workspaces is a promising technology, but until it’s available in the EU-WEST-1, we’re unlikely to be able to adopt it except perhaps in very niche circumstances.

That’s the thing about Amazon though: Like Apple, when Amazon first release a new feature, it’s tempting to be a little underwhelmed. But then, like Apple, a few months or years later we look back at a now mature technology, and we can’t quite remember when it grew up from a metaphorical spotty teenager with potential, to an essential member of the team.

It’s this ability to start ‘simple’, but then improve and polish their products day in, day out, over and over again that has made both companies the unstoppable juggernauts they now are.

 

Please Rate and Like  this blog.  Share it via the social icons below or via short URL http://bit.ly/1hewn2U

Our readers want to know what YOU think, so please add a Comment. 

Fight in ice hockey 2009It’s been a pretty amazing 48 hours or so in the mega-cloud vendor space. We’ve rather lazily got used to continual price reductions from AWS, but this round of Google vs AWS price reductions are pretty special even given this context.

First Google announced some very big price reductions – it was the storage pricing that really grabbed my attention, at 2.6 cents/GB. But for the majority of workloads the compute costs are the dominant component, and so the 32% reduction in compute costs is probably more significant for many. It’s a minor point, but the Google announcement mentioned “reintroducing Moore’s Law to the cloud“, but Moore’s Law is of course finally running out of steam, e.g. according to Intel it’ll be game over by 2020.

AWS have responded with this, but interestingly seem to be calling time on the race to the bottom, knowing that they have a much more credible enterprise offering than Google I suspect. On S3 they’ve almost matched Google but not quite at 3 cents/GB reducing to 2.75 cents/GB with volume. Perhaps the bit that I’m most excited about is the price reduction of the M3 instance range by a whopping 38% (e.g. an m3.large in US-East is reducing from $0.225/hour to $0.140/hour), given that the M3 range is often our preferred weapon of choice these days. That’s a massive bang for your buck.

The next obvious thing to look out for is what Microsoft do with Azure pricing – the assumption is that they will match AWS as per their previous announcement to “peg” their pricing to AWS. Ouch – imagine being an exec and getting out of bed in the morning to find out that you need to drop your prices by 30-80% across the board!

[ADDED 2nd April - Microsoft have done just that - see announcement on their blog here]

So what conclusions can we draw from all this? Well here are mine:

  1. What’s cheapest today is not necessarily cheapest tomorrow – so optimise costs for the long term, not the short term. OK, if you just want some server or storage capacity for a short time then go with the cheapest I guess, but in reality I’m talking about enterprise workloads and it’s never “a short time” – storage is for life, not just for Christmas :), and the cost of moving between providers might outweigh any benefit. Also, the costs are now so low for some workloads (e.g. if I’m playing around with some feature on AWS) that they are trivial anyway – so convenience and sticking with whatever minimises any usage friction are paramount for me.So rather like when choosing a bank to save your money with, where you might want to go for the savings account with the best long term track record of consistent high interest rates rather than the headline grabbing “bonus” offer – when selecting an IaaS cloud provider it’s their trajectory that matters (and hence their ability to leverage mega-scale). It’s not a great time to be a sub-scale (and sub-scale these days still means freakin’ huge) cloud provider unless you’ve got some specific niche offering…
  2. In general, we don’t recommend buying AWS Reserved Instances (RIs) for longer than a 1 year term. The 3 year term often makes more financial sense at that moment in time, but in reality the prices are dropping quicker in a year than the additional saving. This makes sense really, as AWS virtually created the IaaS market only 8 years ago, so a 3 year commitment is still a lifetime in this market.In fact, now is a great time to buy 1 year AWS RIs as it’ll be a few months (you’d have thought!) until the next round of potential price drops – maybe timed to coincide with the next AWS Re:Invent conference in Las Vegas in November – so you’ll get the maximum saving. An exception to my point here is that sometimes 3 year RIs are useful for projects where the TCO needs to be completely fixed and predictable – i.e. cost predictability for a business case is the primary requirement.
  3. A mild concern about where all this is heading – in my view there’s enough competition in the market at present for it to be healthy (i.e. the consumer is winning at the moment), but there is a risk that all but the most massive cloud service providers are squeezed out and the resulting oligopoly means that prices start to creep up again. You could argue that Microsoft’s price pegging announcement is an early sign of an oligopoly forming – reminiscent of the behaviour of the supermarket sector in the UK (where 4 retailers have 76.5% of the market). We’re a few years away from this risk so I don’t think this should infuence enterprise’s investment and vendor selection decisions today.

We’re loving it – what a great time to be migrating customers’ IT workloads to a cheaper, more agile platform where the price is only going down!

CloudExpo2013 Our LocationWhat does last week’s Cloud Expo Europe tell us about the maturity of the market for cloud services in the UK? As an Amazon Web Services Premier Consulting Partner, Smart421 had a stand in the Amazon Web Services Village which gave us a great opportunity to have numerous customer conversations. Wayne Stallwood, one of the AWS Architects from our internal AWS Practice, supported our sales and marketing staff on the stand, and we compared notes afterwards to draw out the key themes.

First of all, one immediate observation was that people were more openly talking about hosting production/live systems in the cloud, i.e. not just the usual dev, DR and startups. We’ve been at this cloud game for about 4 years now and so it is far from new for us (although as a side note, it was interesting to hear Mark Russinovich from Microsoft Azure saying “the cloud is new” this week) and we started to see this shift at least a year ago if not longer. Some of the presentations at Cloud Expo Europe reflected that, for example with a talk about DDoS hardening etc – very much about live systems. There were lots of questions about performance stability, resources, scalability, reliability etc – again more enterprise-level considerations.

Smart421 stand

Smart421 stand

Balancing this though, it was somewhat alarming that some of the people coming to the stand still wanted to talk about the buzzwords without really knowing what they are…so we had a few openers where it was “so this “big data” thing…what does that do for me?” and if you looked at the name badges it was established enterprise people asking the questions. This tells us that there’s still a huge lump of “educational debt” to overcome in the enterprise space.

I had time to attend a couple of presentations but they were pretty awful – dull vendor pitches. You need to choose carefully at these events as the attendees typically don’t pay to attend, and so the bulk of the funding for the event has to be sourced from vendors, and hence they all get to present. There is always some great content though, you just need to be selective and accept you’ll get a few duds.

Instead, I devoted the bulk of my time to understanding Red Hat‘s direction of travel, especially in relation to OpenStack (as I’m fascinated by the cooperation and competition in this area, e.g. from Mirantis , and I’m interested to see how the delivery of private clouds plays out as enterprises use it as a not always sensible stepping stone to the inevitable destination of public cloud) – although inexplicably they were squirreled away on an upper floor and poorly represented in the online show mobile app and so pretty hard for people to find.  I also took some time to catch up with AWS colleagues old and new – including AWS Technical Evangelist Ian Massingham.

The Cloud Expo Europe event itself was co-located with Data Centre World (just over half the floor space) and Big Data Expo Europe (really just a thinly populated stream within the Cloud Expo event), and it was a bit odd to be wandering around the show floor and then stumble into the “dark side” with vendors trying to pitch cooling, racking and UPS power systems to me. I don’t want to build a data centre, ok, AWS has already taken care of that for me :).

The pure cloud content felt smaller to me that previous years, and so as a final thought – I wonder if this reflects not so much that the cloud market is going off the boil but more the opposite – that it’s now mainstream enough that it’s harder to raise so much interest for events that are riding the latest hype?

flock of migrating canada geese birds flying at sunsetIn yesterday’s blog post I summarised the cloud services broker role with some definiti0ns, and concluded that I do indeed appear to work for one of them – and I suspect many vendors at this week’s Cloud Expo Europe (#cee14) might lead on this from a marketing point of view.

We’re delivering service intermediation/customisation and service aggregation/integration, but one thing we are not really doing (or seeing any demand for) at the moment is dynamic or semi-dynamic workload migration. i.e. it’s not just dev & test any more, these days we are migrating complex production environments customer after customer onto AWS. But we are not providing customers with the means to dynamically move or spread those IT workloads across different cloud providers. It’s certainly something we could do from a technology perspective, and most of our deployments have some hybrid aspect to them.

cee_logo_2014_nowebsiteThe ability to migrate IT workloads dynamically (i.e. at run-time, not at deployment time) is something I sometimes see as a capability under the “cloud broker” banner, but in my view it really just doesn’t make sense – at least not at the moment.

The rate of innovation in the IaaS/PaaS/DaaS market is such that most of the other vendors are playing catch-up with AWS, as AWS continue to differentiate themselves from the following pack. This shows no sign of slowing down over the next couple of years – so the only way a migrated workload is going to work across multiple cloud vendors is if it only relies on the lowest common denominator functionality across the vendors, which is typically basic storage, virtualised compute and connectivity.  Or you have to architect your solution to take into account deployment differences across the cloud providers you intend to use – and be able to effectively monitor and support both – twice the work and complexity, and not something you really want to debug.  Did your load balancing just stop working as you expected – it worked last week…mmm…I wonder if our load balancing configuration behaves exactly the same across all our cloud service providers? And so on…

Even storage – the most commoditised of the building blocks of IaaS (you would have thought) contains some interesting differentiation – not on price any more as Google/Microsoft/AWS are effectively price-matching these days, but on features like access control, archiving to cheaper storage, automated data life-cycle policies etc.

The bottom line is that if you are going to architect your applications so they can run on any cloud service provider, then you can’t easily use any of the good bits and hence your value in migrating to a cloud solution is diminished.  Not ruined, just reduced.

There are now a bunch of brokerage tools out there from vendors that claim to give you this workload migration capability, but what I’ve seen so far is disappointing, e.g. one recent new tool I looked at required a custom template to be created for each cloud provider – so whilst the end user might get a menu choice of “deploy to AWS” or “deploy to Azure” – under the covers you still need a bunch of experts in each cloud service provider’s technology and those experts need to keep themselves abreast of new functionality continually.  You can create an impression of homogeneity, but it’s just a veneer.

In our experience, even in very large enterprise estates (e.g. where we’ve deployed and managed up to 750 AWS instances made up of numerous discrete environments), whilst the IT workloads might be relatively consistent in nature (e.g. there might be a corporate standard for this OS and that application server etc), there is always sufficient variance in each project’s requirements that a completely cookie-cutter approach to environment deployment and self-service just does not work.  Each project needs slightly different software, or software versions, or server specifications, or connectivity requirements etc etc – and the list goes on.  And if they didn’t – well – you’d hope the projects would be merged into a single project if they were so similar in their needs, right?

So – given that it’s taken the IT industry this long to get to the point that “as a service” is really possible, for the next couple of years at least let’s focus less on hiding away the good bits of cloud and the really useful differentiating features of the leading cloud providers, and focus more on actually exploiting them please!

PS If you happen to be attending Cloud Expo catch me on the Smart421 stand or let me know you are there via @SmartCTO

CloudBrokerBadgeThe first Cloud Expo event I attended 2 years ago was striking for the myriad of traditional hosting companies who were cloud-washing their offerings (usually quite blatantly and badly I felt).  Last year what struck me was the myriad of new small vendors selling niche cloud-related product offerings – data transfer optimisation, security products, management products, etc. I wonder what the theme will be this year? It’ll be interesting to see how many vendors are wearing the “I’m a cloud brokerage” badge at this week’s Cloud Expo.

Whilst I was at AWS’s Re:invent conference last November, one of the guest speakers at the partner day was Tiffani Bova, Distinguished Analyst from Gartner. Part of her presentation covered the topic of cloud brokerage, something Gartner have been talking about for quite a while, and something Tiffani and I had some debate about afterwards.

I must admit, it took me a while to wrap my head around the concept of cloud brokerage, partially as the pushing of the term was coming more from the analyst community than the rest of the cloud industry. Williams Fellows from 451 Research refers to this as “…a ‘cloudemic’ of firms now stamping ‘cloud broker’ across their service portfolios”. Tiffani’s view was that 90%+ of the AWS partners in the room (including Smart421) were brokers. It’s such a broad definition, e.g. Gartner’s definition is

Cloud services brokerage (CSB) is an IT role and business model in which a company or other entity adds value to one or more (public or private) cloud services on behalf of one or more consumers of that service via three primary roles including aggregation, integration and customization brokerage.

The great thing about definitions is that you can never have enough :). Way back in mid 2011 NIST published the following definition of a Cloud Broker…

NIST CloudBrokerageDefinition

The default view taken in society in that anyone with the title “agent” (aka broker) is looked down upon – e.g. estate agent, recruitment agent etc :). But by this definition I guess we’re all brokers in one way or another, even if it’s just combining toast and scrambled eggs to make the kid’s breakfast in the morning (aggregation).

Looking at what Smart421 delivers for our customers – we integrate different cloud and non-cloud services, we design and implement complex cloud environments and we add a 24×7 ITIL-based service management capability on top including ongoing capacity and cost optimisation. We also add value by handling access management, enhanced security, managing billing complexities and bringing new market innovations to our customers (as cloud service providers like AWS are still innovating and releasing functionality at an amazing rate, too fast for customers to keep up generally).  I guess that means I get to wear the badge too!

In a blog post tomorrow I’ll talk some more about one of the oft-touted cloud brokerage promises – that of dynamically migrating IT workloads across cloud providers (closely related to the dreaded phrase “bursting into the cloud”), and of making deployment-time decisions about which cloud service provider to use…and why I don’t believe in it.

PS If you happen to be attending Cloud Expo Europe this week, catch me on the Smart421 stand or let me know you are there via @SmartCTO

Getting ready for the 2014 conference season, it struck me that the technology revolution has moved from “E” to “D” – by that I mean Digital has replaced Electronic (as in E-Commerce) as the new “must have” conference title.

I remember back in the 1970’s when Digital was replacing Analogue – with the mass introduction of digital watches and calculators – so it amuses me to see it re-cycled. Obviously the generation adopting the word today don’t see the irony in it – nor do the companies inventing Digital divisions.

The serious point is that we’re struggling to articulate the impact of disruptive change on many axis simultaneously. Led by Cloud and Mobile and closely followed by Social Media, Big Data, the need for a secure on-line Identity and even ‘wearable’ technology (back to my old digital watch again) how do enterprises encapsulate the change ?

Cloud is probably the easiest to grasp as it is the most mature and is already releasing its technology potential, but now it needs to be recognised as commercial disruptor – it has already impacted traditional hosting vendors and subjugated “lock-in” contracts they felt were safe. Cloud provides not only a natural ‘leap frogging’ for new entrants into markets by reducing up front set up costs, but can also be a defensive strategy for those businesses trying to adapt to meet rapidly changing customer expectations and behaviours.

Mobile is a key driver changing behaviour – where the acceleration of mobile and tablet (e.g. non-PC) platform adoption is changing the location of the commercial interaction with customers. The mobility of smart phones and tablets has released the consumer from a seat at the desk (office or home PC) and consumers are “inviting” enterprises onto their commuter trains or into their sitting room as they ‘browse in the morning’ and ‘buy in the evening’.

Social Media has benefited from this informal interaction and given access to every review, post, tweet and blog – allowing research ahead of an eventual instore or online purchase – and making C2C communications the primary channel for feedback. Not only do we look up facts at the dinner table using our smart phones but we’re looking at everyone else’s opinion of that new camera, car or city-break as part of the selection process.

All that “opinion” needs a home and – adding it together with all the data produced from location tracking, monitoring and automated machine to machine communication – we have the exponential growth in the volume of data. Then you need tools and techniques to analyse that data (back to Cloud again).

Consumers are also demanding personal interaction which drives the need for Identity – allowing industries to start to drive up the quality and richness of exchanges to enhance customer experience.

So finally I come back to my original question – is my old Digital watch trendy again ?

Please Rate and Like this blog. Share it using the social icons below or via the short URL http://bit.ly/1boh2B7

Our readers want to know what YOU think, so please post a Comment.

Back in September last year I was one of the presenters at an event run by Qualitest where the focus was on big data – the possibilities, implications, testing ramifications etc.

I focused on how to make it real in your organisation and went through a Smart421 case study of the discovery and productionisation phases of a big data initiative, and the run cost implications when using an AWS Elastic MapReduce-based Hadoop infrastructure to run pig queries.  I’ve been meaning to share the YouTube content but Christmas got in the way :), so here it is…

As part of this event I gave a short 6 minute interview on the subject and my perspective on it…

 

…and here’s the presentation itself…

View from the Millbank Tower, London

As part of my focus on developing Smart421’s API strategy, last week I attended Mashery’s one day Business of APIs (BAPI) conference Altitude London,  29th floor of the Millbank Tower on the banks of The Thames.

I’ve been to a fair few events in the last couple of years, from the London AWS enterprise event Robin mentioned in his post this morning to the Gartner AADI conference in May.

The BAPI event had a very different flavour. Although it was a Mashery event, their presence was mainly facilitatory. There was no heavy sales pitch, and no compulsory product demo. The other thing missing was sponsor stands, which was interesting… No need to run the gauntlet with marketing folks to secure your lunch.

What was available in spades was great quality discussion, both from the presenters, and one on one with attendees. The conference was really focussed on the business of building and consuming APIs.

I particularly enjoyed Creating Success / APIs Changing Business from Kevin Flowers, CTO, Coca-Cola Enterprises. Kevin talked about the challenges of launching APIs in a large business. CCE have built a series of internal APIs for managing everything from procurement and finance to sales and service.

This fits well with our internal discussions within Smart421. We think that for many of our customers, the future of integration lies in building and consuming APIs internally first, and then selectively exposing these (or APIs derived from these) to the outside world. Test, learn, do, with the test and learn occurring mainly internally.

TomTom’s Peter Moeykens talked about their experience of building a public API around TomTom’s core map, navigation and point-of-interest assets. He talked about the journey TomTom went on, the (inevitable) missteps made, and the ‘lightbulb moment’ that was the realisation that APIs need to be thought about as products. Again, this is a view I share strongly, even for internal APIs – without taking that outside in view of the world, your APIs will be lifeless. Peter talked about how they sold APIs internally (Answer: demos), and how they sold them externally, and perhaps more to the point, how they got their sales guys to sell them externally (Answer: showing them the kinds of applications the APIs would allow other people to build). Really great, practical advice.

There were a bunch of other great talks too, the slides for which are available on the BAPI website, with videos on the way for the most part. A few of these were only tangentially related to APIs, but were none the less well worth an hour of my time: David McCandless of Information is Beautiful fame talked about data visualisation in a highly amusing talk – well worth a watch when the videos arrive. While Paull Young of charity:water made me laugh – and then damned near cry – with his talk about the work that C:W has been doing on its digital strategy (a topic I’m hoping to talk more about here over the coming months).

So, in summary, a really great event. I’ll definitely be going again next year given a chance, and would encourage anyone interested in this space to go along as well. If you do, hang around a while afterwards – plenty of high quality discussion happened after the main event was over.

Please Rate and Like this blog.  Our readers always like to know what YOU think, so please leave a Comment.

Follow

Get every new post delivered to your Inbox.

Join 1,084 other followers