Per minute IaaS billing – not #1 on my wish list

An interesting announcement from Google at their recent I/O conference was of per minute billing for virtual machine capacity. As we work with AWS a lot, cloud billing is a subject close to my heart – and so this caught my attention. On the surface, per minute-based billing is attractive as there is some inherent wastage in the per-hour model used by AWS, Microsoft Azure etc. When estimating likely AWS usage charges for customer engagements (using the excellent AWS online calculator, which says it is still in beta testing but is actually rock solid), we take great care with the assumptions made about the number of hours that instances will be running for – the classic example of this is for dev/test environments, e.g. it’s quite easy to assume 5 days per week at 10 hours day. What we’ve found over time is that because customers and their development staff have typically been brought up on a diet of inefficient server use (i.e. make a guess what I need, add some capacity contingency on top, pay for them upfront, it’s a sunk cost, so I don’t care about how efficiently I use it), then there is not a strong culture of turning off environments when not in use etc. Also, we control dev/test environments using our SmartSentinel cloud management tooling – and you need to allow a few minutes for instances to startup/shutdown to ensure you don’t fall into an additional hour of cost (especially Windows instances ).

So per-minute billing is attractive as it just cuts down on some over-billing when we “spill” into another hour of usage. But – and this is a big but – the logistics of IaaS billing are already complex enough that I don’t really want it to become more complex. We manage cloud billing for a number of customers, and in a 30 day month we have 720 distinct hourly measurement points where virtual machine usage charges are accrued (to keep it simpler – I’m ignoring other usage based charging, e.g. for storage etc, here). Even with this level of data, validation, reconciliation and invoicing of charges is already very complex. If that became 43,200 measurement points in a month, I think it would tip our finance team over the edge . The complexity stems from the fact that AWS have some really attractive sophistications to their charging model – we like these features and don’t want to lose them, e.g.

• the ability to reserve instances over a 1 or 3 year time period, i.e. make a commitment and share the cost advantage with AWS
• the ability to choose 3 different types of reservation based upon likely usage levels, e.g. 100% on all the time, or rarely on (e.g. for a DR scenario)
• the ability for a customer to get the benefit of their reservations across their various AWS projects/deployment, e.g. if across your AWS estate on average you always have 5 m1.large instances running, but no individual project has them running all the time, you can still reserve the instances and get all the price advantage as the reduced per-hour cost is shared across the entire estate
• volume discounts
• ..and that’s before I even get to spot pricing!

These pricing model sophistications are real differentiators and allow a much more tailored cost model for specific customer deployment scenarios – and I think they are more important than per-minute metering of usage. It’ll be interesting to see if AWS or Azure follow the Google lead (as tends to happen with IaaS pricing between the big boys). Cloud billing truly is becoming a big data problem – if this carries on we’ll need to run up an on-demand AWS EMR Hadoop cluster to do billing reconciliation …

 

The Secret to Amazon’s Success

Jeff Bezos
Photo by John Keatley, Seattle’s leading photographer keatleyphoto.com

Every time I hear this story, it makes me smile. From Kim Lane over at API Evangelist:

[…] one day Jeff Bezos issued a mandate, sometime back around 2002 (give or take a year):

• All teams will henceforth expose their data and functionality through service interfaces.
• Teams must communicate with each other through these interfaces.
• There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
• It doesn’t matter what technology they use.
• All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

The mandate closed with:

Anyone who doesn’t do this will be fired. Thank you; have a nice day!

Assuming for the moment that this is true, the thing that makes me smile here isn’t the closing rhetoric. What Jeff described here is pretty well everything you need to know about successful SOA.

Look at the wording again. “All teams”. He didn’t say “all systems” or “all services”. Technology isn’t [the most] important. People are.

By focussing on teams rather than technology, Jeff ensured that Amazon’s embryonic SOA was business aligned. One, simple decision was all it took. Well, that and ten years of concerted effort of one of the brightest engineering teams on the planet.

 

Mobile Transport Ticketing standards

Tickets please.
Photo: Rail Technology Magazine / Progressive Media Group

The pain you have to go through to travel abroad has always bugged me. I went on a Swiss walking holiday last year and had to buy our rail passes months in advance and there were so many options and routes from various airports that my wife and I burned many an hour discussing alternatives. How much easier would it be if there was an easy way to just buy a ticket from Ipswich to Wengen, Switzerland via any of those routes with your selected airline? Then choose the trains to fit your needs.

That was the dream of the OSPT Alliance and their ticketing interoperability initiatives. The UK transport industry, Department of Transport and the Association of Train Operating Companies (ATOC) instead went for the ITSO standard as national public transport Smart ticketing technology started to come into reality.

Now it appears the two organisations are beginning to work together to avoid the need for travellers to use two different formats and move towards common ticketing on Smart card and eventually Mobile phones, e.g. using NFC.

The International Transport Smartcard Organisation (ITSO) are a limited company who manage that standard from Milton Keynes. ITSO ticketing is one of four broad groups of fulfilment used by national rail. The others are paper tickets, primarily Credit Card Size Tickets (CCST), barcode (used for self printing) and Oyster which is the proprietary format card that has been so hugely successful in London. The ITSO members who supply ticketing and the terminals for validation at stations (so-called POSTs) and by handheld terminals on trains sign up to a code of practice for interoperability and security.

The OSPT Alliance defines the CIPURSE open standard, based on a number of contactless and Near Field Communications (NFC) specifications and it appears to be firmly aiming at the mobile App market. The V2 CIPURSE Mobile spec is published and available to members for evaluation.

We’re not sure exactly how closely these organisations will be working together. The press release today (reproduced below) mentions becoming members of each others’ committees and leveraging complementary aspects. It probably means there will be some kind of integration abstraction layer in the card or network back to the various back-end systems. It probably also means more complexity and work for apportionment and settlement systems run by people like our customer who spoke at the recent AWS Summit ”ATOC Rail Settlement Plan“. In some ways it makes apportionment easier as they should soon be able to track exactly where a customer using mobile ticketing travelled rather than apportion according to estimated volumes who take certain routes as they do today.

Whichever way you look at it, the future world of ticketing is likely to be mobile so what do rail customers think? The quote below comes from the excellent Passenger Focus report on ticketing – available on their website on the subject of buying tickets on their Mobile Phone.

Some respondents had experience of this being a helpful information source that was trusted to identify best tickets or fares for unfamiliar journeys, thereby allaying validity concerns. However all acknowledged that they were unlikely to buy tickets on the phone so these would still need to be purchased elsewhere, meaning that the choice/complexity paradox can only be partially overcome through this channel.

The full emailed press release from the OSPT Alliance is reproduced below:

The Open Standard for Public Transport™ (OSPT) Alliance and ITSO Ltd., the organization responsible for the UK national specification for smart ticketing, today announced they have agreed to participate as members in each other’s organizations and to explore ways they can work together to promote the use of open security standards in public transit for smart ticketing and electronic fare collection systems.

Through their shared commitment to the use of open standards, these two leading public transit standards bodies intend to leverage the complementary aspects of their standards and ecosystems and discuss how they could be combined to create solutions that would be mutually beneficial to their respective members.

“We are pleased to have ITSO join the OSPT Alliance as an associate member, and look forward to exploring how our shared vision for the future of open standards in public transit can result in a mutually beneficial relationship,” said Laurent Cremer, executive director for the OSPT Alliance. “ITSO is an established, recognized player in smart ticketing, and has developed some key technology we believe would be of great interest to OSPT Alliance members as they deploy fare collection systems based on the CIPURSE security standard.”

The CIPURSE open security standard addresses the need by local and regional transit authorities for future-proof fare collection systems with more advanced security than currently in use. Because it is an open standard, CIPURSE promotes vendor neutrality, cross-vendor system interoperability, lower technology adoption risks, higher quality and improved market responsiveness, all of which result in lower operating costs and greater flexibility for transport system operators.

“We welcome the OSPT Alliance as an affiliate member of ITSO, and look forward to their contribution in helping to ensure that public transport operators throughout the UK can continue to maintain the highest level of security in the smart ticketing systems they deploy,” said Lindsay Robertson, chief executive officer of ITSO. “We believe that by working with the OSPT Alliance, ITSO will be better able to supply its members with a more diverse set of card products, including AES-based products, which is a solution the OSPT Alliance can deliver off the shelf in the form of CIPURSE.”

About ITSO

ITSO Ltd. is the non-profit distributing organization that oversees the ITSO Specification for smart ticketing in the UK. ITSO helps its members to set up and run ITSO-compliant smart ticketing schemes, tests and certifies smart ticketing equipment to ensure it meets the ITSO standards and ensures the ITSO Specification is up to date and fit for purpose. ITSO operates the ITSO Security Management System (ISMS), a secure key management and distribution system specifically developed to enable ITSO-compliant smart ticketing systems to be set up.

About the OSPT Alliance

The OSPT Alliance is an international association chartered to define a new open standard for secure transit fare collection solutions. It provides industry education, creates workgroup opportunities and catalyzes the development and adoption of innovative fare collection technologies, applications and services. The OSPT Alliance was founded by leading technology companies, and membership is open to technology providers, transit operators, consultants, solution vendors, government agencies and other stakeholders in the transit ecosystem. For additional information, please visit www.osptalliance.org.

 

Not even standing room only at the London AWS Summit

As the leading enterprise AWS Partner in the UK  ;o)  Smart421 had a big presence at the London AWS Summit last week (23 April).

Several of our customers also attended and one of them, Steve Howes, Chief Executive of Rail Settlement Plan part of Association of Train Operating Companies – ATOC) presented as part of the 2nd keynote, and very kindly mentioned us.

In fact, we were referenced thoughout the day, sometimes in unexpected ways. For example, within the opening five minutes of the first keynote, Smart421 was name-checked by Werner Vogels, CTO at Amazon.

Steve Howes of RSP takes to the stage in front of 1,200 delegates

As per the recent Las Vegas event, it was a bit rock’n'roll in the keynotes, with music by Foo Fighters playing over the PA whilst we were waiting for the queuing hordes to make their way through the registration bottleneck and into the venue.  Vogels himself appeared on stage to the sound of Nirvana pumping out of the speakers.

What struck me was the size that this event had become, a significant increase on the 2012 AWS Summit.

In the afternoon, the event split into separate streams. I stuck to the more involved sessions that were digging into the specifics of particular service releases such as Amazon Redshift and Amazon DynamoDB.

As you can see from the photo above, there was barely sitting room only in some of these techy sessions, let alone standing room.

It made me think – would conference attendees be prepared to sit on the floor for any vendor, especially in the corner of a crowded room? There must be few other vendors, if any, that have that kind of pulling power right now.

These are exciting times. Opportunities for enterprises to benefit are enormous.

And this event confirmed my reflections that we’re reaching a tipping point in the market; adoption by the “early majority” in the technology adoption lifecycle is really visible and is happening – albeit with different market sectors arriving at very different stages.

Our conversations continued all day long over at the Smart421 stand, where we showcased three of our customer engagements:
Disaster Recovery on the AWS Cloud for Haven Power,
Big Data Analytics on the AWS Cloud for Aviva / Quotemehappy.com,
and Service Transition to the AWS Cloud for ATOC.

Frankly, we wanted to showcase even more but there just wasn’t the space.

Please Rate and Like this blog.  If you can, please leave a Comment.

 

Third Generation Outsourcing – SIAM

It was great to see National Rail Enquiries (NRE) win an award at the European Outsourcing Association Awards in Amsterdam last Friday (26 April).

In recognition of their SIAM outsourcing strategy (Service Integration and Management), NRE won the award for Best Multi-sourcing Project of the Year , beating strong category finalists 60k and Centrica (Centrica won this category in 2012).

Smart421 is pleased to be a large part of that initiative, performing the Managed Services element on top of an AWS Cloud platform for several key NRE applications.

As customers struggle with the chains of traditional SI relationships, Smart421 is providing agile delivery and innovation methods in the IaaS world.

Many analysts see this as “third generation outsourcing” and a change for good – and so do I.

 

Please rate and Like this blog.  If you can, please leave a Comment.

 

SyncFocus on Mobile – Alloys are extra

Which Essex boy doesn’t want Titanium Alloys?
Photo © Maxpro Dreamstime Stock Photos & Stock Free Images

Introduction to Appcelerator

I’ve had my eye on Appcelerator and their Titanium mobile development platform product for some time. So I grabbed the opportunity to venture north of the [Suffolk] border to the ”SyncFocus“ a SyncNorwich spin-off event that was held in the Garden House Pub, upstairs room, in Norwich last night (20th March).  So, reflecting on my rough notes, I’m posting this blog with a few random thoughts thrown in and some links to further information for those interested in javascript-based mobile developments.

This SyncFocus featured as guest speaker non other than Boydlee.  For those who haven’t heard of Boydlee, he is a mobile application development guru and specialises in the Titanium platform who, like Beyoncé, is now so famous he doesn’t need to use his surname .

My “executive summary” is that Titanium is a serious productivity tool for people who can write “well written” javascript code. It is in competition with the HTML5-based approach of PhoneGap, and does not (yet) have as many surrounding services as IBM Worklight which is squarely aimed at the big Enterprise market and is not as broad in its coverage of operating systems and devices (yet). It does however, have a thriving developer community and a very attractively priced (i.e. “FREE” cloud service). It does appear to be very easy for a moderately skilled javascript developer to create apps but it’s probably just as easy to create a horrendous mess so the newest approach is to supply a framework as the well-known MVC pattern through “Alloys” – which Essex boy doesn’t want Titanium Alloys on their racer?

Main points:

• Titanium is purely JavaScript library coding against the Titanium api, the platform compiles the source to native Objective-C or Java code for the iOS and Android Apps and can easily be linked with the Xcode or Android SDK to complete the builds.
• Links with emulators for iPhone, iPad and Android
• Appcelerator has over 400,000 registered developers worldwide (a counter on their home page tells me it is just under 430,000 today)
• The libraries themselves are “Free and open source”, available on GitHub
• However, the Titanium studio is not free, and you can take the Appcelerator Paid support
• An estimated one third of cross- platform apps are built on titanium
• Claims to also support BlackBerry 10, but Windows 8 is not available until later this year

One really nice feature is a very large “Module marketplace” which is mostly compiled binary code aimed at one native operating system or the other that can be downloaded and incorporated into the build – for example something to integrate to native libraries, plugins, e.g. Barcode scanning

A lot of those modules also have their source code on GitHub, building the community even more and improving the eco-system. This appears to be thriving and also includes some javascript extensions as well as the compiled native iOS and Android libs.

Appcelerator Mobile Service (ACS)

ACS is an interesting move to give a free level of Mobile (application) Back-end as a Service (MBaaS) to developers of services, running on AWS.

This includes 20GB storage, 5m push, 5m api and 20 pre built common services, including key-value pairs (similar to AWS Simple DB), Ratings, Geo services, media, including thumbnails, Social integration (Facebook) and storage of custom objects, which I think was based on Mongo DB as a service. There is certainly capability to link directly with Node and Mongo backends and simple stuff like SMTP integration (again similar to AWS’s Simple Email Service “SES”).

Coding practices

Boydlee recommended looking into Common.js best practice for relative newbies to Javascript. Other frameworks are available, like backbone, require.js and one of his blog posts has recommended the javascript patterns book by Stoyan Stefanov.

There are also free books online available on styling, including best practices to cascade for tablets so you don’t just get an enlarged phone experience that sucks.

The CSS approach also over-rides for different devices, so it’s not always best to stick with pure javascript for everything, however, for those who really are blinkered by javascript there is a CSS to JSON converter library.

Boydlee’s books are on PACKT or there are kindle versions. Boydlee’s best practices book is one I will be getting for sure.

Finally, the newest developments in Appcelerator will probably be using Alloy which includes full Model View Controller (MVC) frameworks (again, all built on well-known javascript frameworks) that claims to speed up App development by another order of magnitude.

Comparing it with the IBM Mobile First offering the coverage of mobile O/S is much wider in Worklight (Windows 8 and BB 10 were available very soon after launch), the adaptor environment, the upgrade management and enterprise app centre seem more developed and you have the whole extra wrappers of security, analytics and so on within IBM’s Mobile offering. The customer list on Appcelerator’s website is mighty impressive for consumer-oriented Apps but the ability to integrate with Enterprise back-ends is less of a focus, which is why in our case we’re focusing on IBM for Enterprise mobility but Appcelerator is certainly worth considering.

Please Like and Rate this post.  If you can, please leave a Comment.

 

Big Data more than just analytics

Enterprises that are thinking about big data need to understand that it isn’t just about analytics of vast amounts of data but also how that information is stored, Amazon CTO Werner Vogels said during a keynote at a recent Cebit trade show.

His “Data without limits” speech encouraged enterprises to think about the big picture, and he also presented a blueprint for how Amazon’s cloud can be used to take out some of the pain of implementing Big Data systems.

“Big Data is not only about analytics, it’s about the whole pipeline. So when you think about Big Data solutions you have to think about all the different steps: collect, store, organise, analyse and share,” said Vogels.

Amazon itself has been doing Big Data and analytics for a long time to try to target customers and convey relevant recommendations. When mistakes have been made it’s because there isn’t enough data to back up a recommendation.
Amazon Web Services offers a number of services that can help an enterprise collect, store, organise, analyse and share data:
⁃    AWS Direct Connect allows an enterprise to establish a dedicated network connection from a customer to Amazon.
⁃    AWS Import/Export, which allows an enterprise to send portable storage devices to Amazon, which are then uploaded to Amazon’s cloud storage.

Other services that are also used for big data include:
⁃    Amazon’s Simple Storage Service
⁃    DynamoDB NoSQL database
⁃    Apache Hadoop-based Elastic MapReduce

A new service from Amazon, Data Pipeline can orchestrate the processing of data using data-driven workflows and built-in dependency checking in order to collect, store, organise, analyse and share, large data. Amazon designed the Data Pipeline service to automate the process of parsing large sets of data.  After processing the data, the results can be transferred to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon Elastic MapReduce (EMR), and the Data Pipeline service is integrated with the Amazon Simple Notification Service (Amazon SNS) to send notifications of any failures during processing of the data.

Please Rate and Like this blog. If you can, please leave a Comment.

 

Survival quip delivers powerful lesson on technological change

Photo © Ciprian.d Stock Free Images & Dreamstime Stock Photos

As you already know from the blog 5 Dec 2012, as we were preparing to tell  the world about our Big Data capabilities, our CTO Robin Meehan coined a cracking one-liner:

“It’s not about survival of the fittest – it’s about survival of the best informed”

We liked it so much we decided to include it on our go-to-market materials.

Since then, we’ve witnessed some upsides such as how our customers have decided to deliberately ’think outside of the box’ to better understand their portfolio of brands and be better equipped to attract customers without canabalisation of other lines of business.

A great case in point would have to be Quotemehappy.com which is already taking advantage of the analytical power in cloud computing with Big Data on AWS to understand its brand and build its business.

Aviva launched Quotemehappy.com in August 2011. With a national reputation and strong brand in the general insurance sector at stake, they wanted to know how their multi-touchpoint cross marketing activities impacted each brand in their enterprise portfolio.

Smart421 was invited to assist.  Smart421 architected the Cloud instances using Amazon Web Services (AWS) and developed the algorithms needed to maximise the power of the customer’s own data and the Big Data analytical environment. This gave the business a level of insight not previously possible with traditional on-premise business intelligence tools and techniques.

And the customer was kind enough to go on record about what we had been able to do.

“Smart421’s Cloud architects gave us a head start on making Big Data real for us, including how business insights are really delivered, what the costs really are, and how the technology really works in our context. Their output contributes to how we differentiate ourselves in a crowded market.”

Keith Misson, Operations Director at Quotemehappy.com (an Aviva company).

Naturally, we asked if we could feature this on our website because it evidences the transformation effect of what a well architected IT strategy can deliver for a business.

(Why not share this case study with your colleagues and friends using short URL http://bit.ly/Wby5Y6 )

Big Data is a good example of how technologies developed for one use have been deployed for an altogether different use. I think that Robin’s original quip on survival has actually gone on to deliver a powerful lesson on technological exaptation.

 

Please Rate and Like this post, If you can, please leave a Comment.

 

AWS Toolkit CloudFormation Editor – Thanks Santa

In my previous blog post on using AWS CloudFormation to provision a Centos based environment I mentioned how at first the JSON syntax used within CloudFormation templates can be a little daunting, especially for people with limited scripting experience.

At the time of writing the blog I was using a combination of   Notepad++ with the JSON Viewer Plug-In to create and maintain my CloudFormation templates. One of the problems I had with this approach was that the templates are treated as pretty basic text files by Notepad++ and the JSON Viewer Plug-In only checks the base JSON syntax, so there is no validation of the objects defined in the JSON to ensure that the correct properties are being creating to allow the template to form a valid CloudFormation stack.

As an early Christmas present to anyone working with CloudFormation AWS recently announced the introduction of a new CloudFormation editor as part of the AWS Toolkits for Visual Studio and Eclipse. I have now had a chance to download and experiment with the Visual Studio version of this and am really impressed with how much easier it makes the creation and maintenance of CloudFormation templates.

As a .Net developer I have grown used to regularly relying upon IntelliSense features to aid with code creation, particularly when it comes to accessing and updating object properties. The CloudFormation editor provides this for the objects defined within a template as well as code snippets for all the object types you might want to define within your template. This greatly reduces the number of errors caused by ‘finger problems’ that used to occur when creating a template by hand.

The other really useful feature of the editor is the ability to estimate the costs of a stack before creating it. In the past when quoting for customers we have tended to pull together a design, plug the figures into the AWS Simple Monthly Calculator and then once we have received the go ahead provision the environment. With the ability to create costs from a template we are now looking at generating a base template at the design phase and then costing and building from this, which should help with the speed of environment creation and clarity around the prices for altering the design.

Based on our experiences so far  it’s a big thank-you to AWS Santa and his helper elves for the early Christmas present of the CloudFormation Editor and we are all looking forward to further presents from AWS  throughout next year

 

Adventures in AWS EC2 spot land

I was doing some Hadoop demo work last week for a customer and mainly just because I could, I used spot instances to host my Hadoop/pig cluster using AWS’s Elastic MapReduce (EMR) offering.  I thought I’d have a quick look at what the resulting costs were over the few hours I was using it.  I used a combination of small and large instances in the US-East region – m1.small for the master node and m1.large for the core nodes.  Note – these costs exclude the PaaS-cost uplift for using EMR (another 6 cents per hour for a large instance).

In summary – it’s dirt cheap….

What is more revealing is to look at this in terms of the % of the on-demand price that this represents…

So in summary, around an average saving of 90% on the on-demand price!  This is probably influenced by the fact that I was running the cluster mainly during the time when the US are offline.  We tend to get a bit fixated on the headline EC2 cost reductions that have frequently occurred over the last few years, and the general “race to the bottom” of on-demand instance pricing between AWS, Google, Microsoft etc.  Obviously not all workloads are suitable for spot pricing, but what I did here was deliberately bid high (at the on-demand price for each instance type in fact) knowing that this would mean that I was very unlikely to get booted off the instances as anyone bid higher if capacity got short.  As EC2 instance costs are so low anyway, we tend to not worry too much about optimising costs by using spot pricing for many non-business critical uses – which is a bit lazy really and we could all exploit this more.  Let’s do that!