Service Versioning for cheapskates with Feature Flags

Photo: Industrial backdrop by Pilarts Dreamstime Stock Photos & Stock Free Images

I’d like to propose a best practice for rolling out new features in a Service Oriented Architecture (SOA).

Traditionally, when we roll out a major new feature, we often end up causing a breaking change to the service. We’re then faced with a choice: (a) Force all our consumers to upgrade to the new version, and making all our consumers hate us, or (b) continue to support the old version of the service as well as the new, making only our own teams hate us. Suck it up, plan (b) is the better option, but try telling that to the guy having to patch fixes in three concurrent versions of a service.

Now, there are patterns that can help here (more on that another day), but they all still mean more work for everyone.

Also, when we first roll out a feature is exactly the moment we understand it least. We’ve got absolutely no idea how people will use it, nor whether it will even turn out to be useful. By baking the feature into a new major version of the service, we’re taking all our options away. The feature will be hard to remove if we decide it isn’t useful, and if we want to change how it works, we’re back into a major version upgrade again.

To my mind, good engineering is largely about keeping your options open. It’d be nice if we can try a new feature with a subset of consumers first, iterating quickly with just that subset, gradually adding more consumers as we get more confident.

Enter the Feature Flags pattern. Feature flags allow you to turn features on an off at a moment’s notice. At its most basic, a feature flag just turns a feature on or off for everyone at once, but the idea is often extended to allow turning on features for specific users, or collections of users. This allows you to roll out a new feature to consumers gradually, over an extended period.

So, here’s the proposal:

• Allow consumers to pass a set of feature flags dictating which features they’d like enabled in the service.
• Whenever you build a major new feature that would otherwise cause a breaking change, only enable it when the feature flag is passed.
• If appropriate to your environment, control access to feature flags like you would to any other resource – e.g. you might want to restrict access in the early days to just a single consumer, making it easier to iterate.
• Once we’re comfortable with a feature, it becomes publicly available – i.e. anyone can toggle the flag.
• Every so often (e.g. once every couple of years), create a new major version of the service, refactoring it to include popular, battle tested features by default. Also, take this as an opportunity to clean out the cupboard and abandon any features that aren’t well used.

What do you think? Comments and thoughts very welcome…

 

Please remember to Rate and Like this post.  If you can, please leave a Comment.

 

The Secret to Amazon’s Success

Jeff Bezos
Photo by John Keatley, Seattle’s leading photographer keatleyphoto.com

Every time I hear this story, it makes me smile. From Kim Lane over at API Evangelist:

[…] one day Jeff Bezos issued a mandate, sometime back around 2002 (give or take a year):

• All teams will henceforth expose their data and functionality through service interfaces.
• Teams must communicate with each other through these interfaces.
• There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
• It doesn’t matter what technology they use.
• All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

The mandate closed with:

Anyone who doesn’t do this will be fired. Thank you; have a nice day!

Assuming for the moment that this is true, the thing that makes me smile here isn’t the closing rhetoric. What Jeff described here is pretty well everything you need to know about successful SOA.

Look at the wording again. “All teams”. He didn’t say “all systems” or “all services”. Technology isn’t [the most] important. People are.

By focussing on teams rather than technology, Jeff ensured that Amazon’s embryonic SOA was business aligned. One, simple decision was all it took. Well, that and ten years of concerted effort of one of the brightest engineering teams on the planet.

 

Mobile Transport Ticketing standards

Tickets please.
Photo: Rail Technology Magazine / Progressive Media Group

The pain you have to go through to travel abroad has always bugged me. I went on a Swiss walking holiday last year and had to buy our rail passes months in advance and there were so many options and routes from various airports that my wife and I burned many an hour discussing alternatives. How much easier would it be if there was an easy way to just buy a ticket from Ipswich to Wengen, Switzerland via any of those routes with your selected airline? Then choose the trains to fit your needs.

That was the dream of the OSPT Alliance and their ticketing interoperability initiatives. The UK transport industry, Department of Transport and the Association of Train Operating Companies (ATOC) instead went for the ITSO standard as national public transport Smart ticketing technology started to come into reality.

Now it appears the two organisations are beginning to work together to avoid the need for travellers to use two different formats and move towards common ticketing on Smart card and eventually Mobile phones, e.g. using NFC.

The International Transport Smartcard Organisation (ITSO) are a limited company who manage that standard from Milton Keynes. ITSO ticketing is one of four broad groups of fulfilment used by national rail. The others are paper tickets, primarily Credit Card Size Tickets (CCST), barcode (used for self printing) and Oyster which is the proprietary format card that has been so hugely successful in London. The ITSO members who supply ticketing and the terminals for validation at stations (so-called POSTs) and by handheld terminals on trains sign up to a code of practice for interoperability and security.

The OSPT Alliance defines the CIPURSE open standard, based on a number of contactless and Near Field Communications (NFC) specifications and it appears to be firmly aiming at the mobile App market. The V2 CIPURSE Mobile spec is published and available to members for evaluation.

We’re not sure exactly how closely these organisations will be working together. The press release today (reproduced below) mentions becoming members of each others’ committees and leveraging complementary aspects. It probably means there will be some kind of integration abstraction layer in the card or network back to the various back-end systems. It probably also means more complexity and work for apportionment and settlement systems run by people like our customer who spoke at the recent AWS Summit ”ATOC Rail Settlement Plan“. In some ways it makes apportionment easier as they should soon be able to track exactly where a customer using mobile ticketing travelled rather than apportion according to estimated volumes who take certain routes as they do today.

Whichever way you look at it, the future world of ticketing is likely to be mobile so what do rail customers think? The quote below comes from the excellent Passenger Focus report on ticketing – available on their website on the subject of buying tickets on their Mobile Phone.

Some respondents had experience of this being a helpful information source that was trusted to identify best tickets or fares for unfamiliar journeys, thereby allaying validity concerns. However all acknowledged that they were unlikely to buy tickets on the phone so these would still need to be purchased elsewhere, meaning that the choice/complexity paradox can only be partially overcome through this channel.

The full emailed press release from the OSPT Alliance is reproduced below:

The Open Standard for Public Transport™ (OSPT) Alliance and ITSO Ltd., the organization responsible for the UK national specification for smart ticketing, today announced they have agreed to participate as members in each other’s organizations and to explore ways they can work together to promote the use of open security standards in public transit for smart ticketing and electronic fare collection systems.

Through their shared commitment to the use of open standards, these two leading public transit standards bodies intend to leverage the complementary aspects of their standards and ecosystems and discuss how they could be combined to create solutions that would be mutually beneficial to their respective members.

“We are pleased to have ITSO join the OSPT Alliance as an associate member, and look forward to exploring how our shared vision for the future of open standards in public transit can result in a mutually beneficial relationship,” said Laurent Cremer, executive director for the OSPT Alliance. “ITSO is an established, recognized player in smart ticketing, and has developed some key technology we believe would be of great interest to OSPT Alliance members as they deploy fare collection systems based on the CIPURSE security standard.”

The CIPURSE open security standard addresses the need by local and regional transit authorities for future-proof fare collection systems with more advanced security than currently in use. Because it is an open standard, CIPURSE promotes vendor neutrality, cross-vendor system interoperability, lower technology adoption risks, higher quality and improved market responsiveness, all of which result in lower operating costs and greater flexibility for transport system operators.

“We welcome the OSPT Alliance as an affiliate member of ITSO, and look forward to their contribution in helping to ensure that public transport operators throughout the UK can continue to maintain the highest level of security in the smart ticketing systems they deploy,” said Lindsay Robertson, chief executive officer of ITSO. “We believe that by working with the OSPT Alliance, ITSO will be better able to supply its members with a more diverse set of card products, including AES-based products, which is a solution the OSPT Alliance can deliver off the shelf in the form of CIPURSE.”

About ITSO

ITSO Ltd. is the non-profit distributing organization that oversees the ITSO Specification for smart ticketing in the UK. ITSO helps its members to set up and run ITSO-compliant smart ticketing schemes, tests and certifies smart ticketing equipment to ensure it meets the ITSO standards and ensures the ITSO Specification is up to date and fit for purpose. ITSO operates the ITSO Security Management System (ISMS), a secure key management and distribution system specifically developed to enable ITSO-compliant smart ticketing systems to be set up.

About the OSPT Alliance

The OSPT Alliance is an international association chartered to define a new open standard for secure transit fare collection solutions. It provides industry education, creates workgroup opportunities and catalyzes the development and adoption of innovative fare collection technologies, applications and services. The OSPT Alliance was founded by leading technology companies, and membership is open to technology providers, transit operators, consultants, solution vendors, government agencies and other stakeholders in the transit ecosystem. For additional information, please visit www.osptalliance.org.

 

Not even standing room only at the London AWS Summit

As the leading enterprise AWS Partner in the UK  ;o)  Smart421 had a big presence at the London AWS Summit last week (23 April).

Several of our customers also attended and one of them, Steve Howes, Chief Executive of Rail Settlement Plan part of Association of Train Operating Companies – ATOC) presented as part of the 2nd keynote, and very kindly mentioned us.

In fact, we were referenced thoughout the day, sometimes in unexpected ways. For example, within the opening five minutes of the first keynote, Smart421 was name-checked by Werner Vogels, CTO at Amazon.

Steve Howes of RSP takes to the stage in front of 1,200 delegates

As per the recent Las Vegas event, it was a bit rock’n'roll in the keynotes, with music by Foo Fighters playing over the PA whilst we were waiting for the queuing hordes to make their way through the registration bottleneck and into the venue.  Vogels himself appeared on stage to the sound of Nirvana pumping out of the speakers.

What struck me was the size that this event had become, a significant increase on the 2012 AWS Summit.

In the afternoon, the event split into separate streams. I stuck to the more involved sessions that were digging into the specifics of particular service releases such as Amazon Redshift and Amazon DynamoDB.

As you can see from the photo above, there was barely sitting room only in some of these techy sessions, let alone standing room.

It made me think – would conference attendees be prepared to sit on the floor for any vendor, especially in the corner of a crowded room? There must be few other vendors, if any, that have that kind of pulling power right now.

These are exciting times. Opportunities for enterprises to benefit are enormous.

And this event confirmed my reflections that we’re reaching a tipping point in the market; adoption by the “early majority” in the technology adoption lifecycle is really visible and is happening – albeit with different market sectors arriving at very different stages.

Our conversations continued all day long over at the Smart421 stand, where we showcased three of our customer engagements:
Disaster Recovery on the AWS Cloud for Haven Power,
Big Data Analytics on the AWS Cloud for Aviva / Quotemehappy.com,
and Service Transition to the AWS Cloud for ATOC.

Frankly, we wanted to showcase even more but there just wasn’t the space.

Please Rate and Like this blog.  If you can, please leave a Comment.

 

Third Generation Outsourcing – SIAM

It was great to see National Rail Enquiries (NRE) win an award at the European Outsourcing Association Awards in Amsterdam last Friday (26 April).

In recognition of their SIAM outsourcing strategy (Service Integration and Management), NRE won the award for Best Multi-sourcing Project of the Year , beating strong category finalists 60k and Centrica (Centrica won this category in 2012).

Smart421 is pleased to be a large part of that initiative, performing the Managed Services element on top of an AWS Cloud platform for several key NRE applications.

As customers struggle with the chains of traditional SI relationships, Smart421 is providing agile delivery and innovation methods in the IaaS world.

Many analysts see this as “third generation outsourcing” and a change for good – and so do I.

 

Please rate and Like this blog.  If you can, please leave a Comment.

 

SyncIpswich 2 brought us Continuous Delivery and Windows Azure Cloud

Wow! SyncIpswich’s second meetup and around 80 people crammed into the Eastern Enterprise Hub in the James Hehir Building at University Campus Suffolk. Many of the attendees were working for local behemoths like BT but there were also a good mix of bootstrappers, Start Ups and tech entrepreneurs with all kinds of backgrounds (even spotted a Chartered Accountant).

Organisers Carl Farmer (@CarlFarmer), supported by Anders Fisher (@atleastimtrying) and others have done a great job with SyncIpswich, which we are proud to sponsor. The focus of this meetup was on building software quickly with good practices as well as a nice introduction to the Windows Azure Cloud.

Talk no 1. Continuous Delivery

The first presentation by Chris O’Dell from 7digital (@ChrisAnnOdell) described how Agile practices (CI, Kanban, etc) combined with their architectural evolution to SOA have reduced code to deploy times to half a day  at 7digital.  And, by the sound of it, makes their developers more productive by getting away from “DLL Hell” that used to be the bane of any Microsoft Windows developer’s life towards a loosely-coupled set of services and a public API.

Chris raised some really interesting points around developing small fine-grained service components – not being that familiar with .Net myself this seemed to be similar to what we are doing in the Java world with OSGi and Service Component Architecture. I do like the policy of developing new features on the trunk (no feature branches) but making good use of feature flags rather than old-fashioned branch & merge.

They are also using Git for the code version control and Chris showed the inversion of the classic Unit Test, Acceptance Test, QA triangle. Some in our own organisation are raising question marks about the usefulness of very granular unit tests so the approach taken by 7digital of increasing the number of Unit tests is interesting.

There were a lot of questions from the floor, I was particularly interested in how the small kanban teams (about 6 or 7 members in 5 or 6 teams I think) interact when there are common services. This is a key problem that us SOA architects need to get right to get the best value on services. Feature Flags is something that we’ve also thought about in the context of simplifying application testing by, for example, switching off authentication for functional testing.

It’s great to see a company like 7digital competing successfully with iTunes and Amazon in the digital music space. I’ll be checking out their API (and their JLS back catalogue !) in more detail this weekend.

Talk no 2. Starting out in Azure

The second talk of the night was by Richard Astbury (@richorama) of Two10 Degrees ( @two10degrees). Richard gave a nice introduction to Cloud computing and in particular using Microsoft Windows Azure, showing a picture of a MS data centre under construction, which was something I haven’t ever seen before. I think it really brought home the sheer scale and commodity nature of the Cloud and these facilities being full of containers of kit that is just thrown away or recycled when it stops working.

Building a website on Windows Azure from scratch can use a few main pre-canned routes like the obvious “Website”, “Virtual Machine” and “Cloud Service”.

And it now includes a “Mobile Service” which is of particular interest to me. Sadly, I didn’t have time to chat to Richard about this but it’s on my “To Do” list to get a Hello Smartie mobile service up and running. In fairness, Richard did do two masterful demos for Website, including a node.js based site which he even launched from his home computer (a Raspberry Pi no less). As Carl tweeted:

Deploying to Azure from a remote RaspberryPi at home… Impressive stuff from @richorama !

— SyncIpswich (@SyncIpswich) April 25, 2013

Well done to the people of Ipswich for turning out and drinking all the sponsored free beer!
SyncIpswich will run and run.

Please Rate and Like this blog.  If you can, please leave a Comment.

 

Optimising big data processing

When you lift the lid on what is going on in the big data analytics world, the pace of development and innovation is phenomenal. Yesterday I heard some more about the range of optimisations and ongoing development being progressed for Apache Hive and Hadoop itself, and it really reminds me of the pace of innovation and development of other technologies I’ve seen in the past, e.g. relational databases, application servers, web and mobile development platforms etc. Hadoop has only existed for seven years or so and the data analytics revolution that it kicked off is morphing further and further away from its origins – I guess that’s natural selection in operation. Anyone still using WML over WAP?

HortonWorks are a big contributor to the Hive project and the Stinger initiative they are involved in is driving a number of really interesting optimisations and enhancements:

• Further optimising the already optimised RCFile data storage structure to allow queries to avoid I/O for blocks of data held in HDFS that contain no data relevant to the query, or where preaggregated results can avoid I/O (precalculated min, max values etc)
• Optimising data retrieval to best exploit CPU on-chip memory buffers
• Exploiting YARN (“Yet-Another-Resource-Negotiator” – a recent framework and sub-project of Apache Hadoop that facilitates writing arbitrary distributed processing frameworks and applications) and Tez (a new Apache incubator project) to avoid unnecessary HDFS writing and subsequent reading of intermediate results between sequential MapReduce jobs
• Using “always on” Hadoop implementations (aka Tez service) to avoid the heavy penalty of JVM startup costs – which are very significant for otherwise relatively low cost queries
• Enhancing query optimisation for common use cases, e.g. the loading of dimension tables (which are relatively small in comparison to the fact table) into memory on each node in the cluster to accelerate common queries against a data warehouse star schema

Olivier Renault from HortonWorks presented some early data at yesterday evening’s London Hive meetup that showed that they’d seen Hive query performance times drop by a multiple of up to 35-40 using a combination of some of the above optimisations. So the Stinger initiative objective of a 100x performance improvement seems feasible, which really would be a transformational achievement. When most people experience a pig/Hive query on a Hadoop cluster for the first time there is a rather “oh – that was slow for a simple query…” reaction – and the strong drive to move Hadoop processing closer and closer to being able to support interactive query use cases is causing some interesting overlaps. For example, Cloudera’s Impala takes a different approach to performance optimisation where all data is loaded into memory – so it can provide blinding fast query performance but won’t ultimately scale as far as Hive (see here for more detail on this).

To be honest, for most of us mere mortals (who don’t work for Facebook, Yahoo! or Google), the tools are now already out there to handle 99% of the use cases we need – and we can see from the above work that they are improving at a rate such that supporting interactive queries for multiple users on very large corporate datasets is becoming a reality, so our customers will just expect this in the future without question.

Update – 10th April – A similar set of slides to the ones Olivier presented are now available here