Survival quip delivers powerful lesson on technological change

Photo © Ciprian.d Stock Free Images & Dreamstime Stock Photos

As you already know from the blog 5 Dec 2012, as we were preparing to tell  the world about our Big Data capabilities, our CTO Robin Meehan coined a cracking one-liner:

“It’s not about survival of the fittest – it’s about survival of the best informed”

We liked it so much we decided to include it on our go-to-market materials.

Since then, we’ve witnessed some upsides such as how our customers have decided to deliberately ’think outside of the box’ to better understand their portfolio of brands and be better equipped to attract customers without canabalisation of other lines of business.

A great case in point would have to be Quotemehappy.com which is already taking advantage of the analytical power in cloud computing with Big Data on AWS to understand its brand and build its business.

Aviva launched Quotemehappy.com in August 2011. With a national reputation and strong brand in the general insurance sector at stake, they wanted to know how their multi-touchpoint cross marketing activities impacted each brand in their enterprise portfolio.

Smart421 was invited to assist.  Smart421 architected the Cloud instances using Amazon Web Services (AWS) and developed the algorithms needed to maximise the power of the customer’s own data and the Big Data analytical environment. This gave the business a level of insight not previously possible with traditional on-premise business intelligence tools and techniques.

And the customer was kind enough to go on record about what we had been able to do.

“Smart421’s Cloud architects gave us a head start on making Big Data real for us, including how business insights are really delivered, what the costs really are, and how the technology really works in our context. Their output contributes to how we differentiate ourselves in a crowded market.”

Keith Misson, Operations Director at Quotemehappy.com (an Aviva company).

Naturally, we asked if we could feature this on our website because it evidences the transformation effect of what a well architected IT strategy can deliver for a business.

(Why not share this case study with your colleagues and friends using short URL http://bit.ly/Wby5Y6 )

Big Data is a good example of how technologies developed for one use have been deployed for an altogether different use. I think that Robin’s original quip on survival has actually gone on to deliver a powerful lesson on technological exaptation.

 

Please Rate and Like this post, If you can, please leave a Comment.

 

Storage Virtualisation – how hard can it be??

Turning disc by Toshiyuki IMAI.
CreativeCommons

With most enterprise arrays now providing the ability to allocate more storage than is physically installed and with data moving transparently between tiers of low cost and high performance with no human intervention – what could possibly go wrong?

For a long time now SAN infrastructure has been at the core of the Data Centre satisfying a need to hoard as much data as could possibly be collected. With the required capacity growing at an exponential rate, it has become common to predict massive data growth when designing services, often calculating growth for two or more years. This scenario leads to large amounts of storage being purchase in anticipation of things to come, by the time two years has passed, some of the attached systems will become legacy as they are superseded by shiny new solutions and some of the systems will just not have grown as expected. Even those systems that have behaved as designed are probably using no more than 70% of the storage as likely as not due to a “fudge factor” arbitrarily thrown into the mix by the architect.

Virtualised storage arrays (each in it’s own way it would seem) allow us, within reason, to allocate as many devices of whatever size we like to as many hosts as we see fit with only a small pool of physical disk. Allowing us to satisfy our storage hungry analysts whilst buying the disk in a phased manner and delaying spend until it is necessary. Add into this the reducing cost per Gigabyte over time and the benefits mount up.

At the same time, the array architecture has developed in such a way that it is harder to facilitate small amounts of physical growth. With larger RAID groups becoming more common and larger disk sizes compounding the issue, traditional SAN allocations became inflexible and for a small requirement, potentially expensive. The smallest building block becomes a RAID Group and for a one Terabyte requirement on a traditional storage array might require the business to grow an array by thirty Terabytes or more. With virtualisation comes the ability to create more devices for allocation without having to have physical storage installed to match. There is no longer a need to grow a system until the written data exceeds thresholds and the larger building blocks are far more palatable when they are shared between all attached systems.

As with everything in life, there is always a flip side. The balance in this case is increased risk; what if the physical pool fills up. At the very least, writes will have to stop but in some cases I/O stops altogether, neither scenario is one that a business will want to contemplate. This is where the need for careful planning and monitoring is essential.

In a simple virtualised environment, physical storage will be configured into Storage Pools and a number of virtual devices created and attached to these pools. As data is written to the logical device it will be written across the physical disks making up the storage pool.

Considerations when designing virtual solutions:

• The number of servers that will be connected – To provision for sufficient connectivity and I/O capability. It is often not available but any additional I/O profiling for attached systems will also be useful.
• The average server allocation size and utilisation – To calculate required space for allocation and physical writes.
• The maximum allocation size –  Large allocations could result in a single server having the ability to fill all available pool space.
• Maturity of the service – Mature systems will require larger initial space with slower growth; new systems may start small and grow into the allocated space over an extended period.
• Performance – Is there sufficient throughput for the attached servers?
• Criticality – There is an associated risk with over provisioning. There are ways to mitigate this and in some cases systems will be too critical to the business to accept any additional risk.
• Mixed Production, Test and development environments – Will the different environments share the same pools or should they be ring fenced?
• Alerting  – What level of alerting is configured? At what level of utilisation do warnings start? Can unexpected write bursts from a single application be identified and highlighted?
• Capacity for growth – Can the pool be quickly expanded?
• Time to deploy – How quickly can the vendor react to increase the Pool size.
• Plan B – Can space be redeployed to production systems?
• Cost reduction per GB – Flexibility may be cited as a benefit but the main driver for virtualised storage is to drive the cost per GB down, on the most part this is achieved by deferred purchase and tiered storage.

Monitoring:

With thin provisioning, monitoring is essential to maintaining a low risk environment.

As a minimum the capacities must be monitored on a regular basis depending on the activity in the environment, the attitude to risk and the subscription and utilisation figures. In most large environments daily should be sufficient or even weekly.

We have found the following capacity metrics to be the most useful and collect on a weekday basis so that we can produce trend analysis to support future forecasts.

For each pool on each array we collect:

• Installed storage
• Used
• Virtual TB
• Allocated
• Subscription
• Nett subscription

Experience shows that the larger environments have a very linear growth both in allocated and written storage and the trending figures provide quite accurate forecast estimates.

Performance figures should also be reviewed but a less regular basis is sufficient – weekly or even monthly.

With virtualisation improving the overall utilisation and ever larger physical disks being installed we see as a result that the I/O profile per disk is also increasing. These performance trends should be monitored and reviewed to anticipate thresholds being reached.

As a result of the increased utilisation and larger physical disks, we are also seeing the number of host connections to front-end ports (fan-out ratio) and the amount of storage allocated to front-end ports both increasing. The host ports should be monitored in the same way as the disks to anticipate performance thresholds being reached.

There is no point increasing the size of the pools to accommodate further allocations if the path to the data is already fully loaded.

This performance monitoring is specific to the capacity planning and does not replace the normal daily performance monitoring and alerting on the storage arrays.

What can be done to prepare for emergencies?

Most virtualised storage solutions have the functionality to preallocate storage. For systems that cannot afford any outage the storage can be preallocated in advance.

Understand the process with the vendors to increase the installed physical capacity.

• How long for the vendor to install new storage from the request?
• Can elements (such as procurement) be prepared in advance of the need or post deployment to avoid delaying the process?
• Can hardware be left on site to reduce hardware deployment timescales?
• Are sufficient resources available to accept additional disk?
  • Free drive bays
  • Cache
  • Power
  • Floor space

Understand the environment.

• What systems are attached that could be sacrificed to maintain critical services?
• Are there device replicas in the pool for non-production purposes that could be released?

These observations are based on simple virtualised pools, most hold true for tiered storage environments but these also bring their own sets of concerns and even more metrics to follow.

Please Rate and Like this blog. If you can, please leave a Comment.

 

AWS CloudFormation (and Chef) on Centos

If you have any experience of supporting large scale infrastructures, whether they are based on ‘old school’ tin and wires, virtual machines or cloud based technologies you will know that it is important to be able to create consistently repeatable platform builds. This includes ensuring that the network infrastructure, ‘server hardware’, operating systems and applications are installed and configured the same way each time.

Historically this would have been achieved via the use the same hardware, scripted operating system installs and in the Windows application world of my past the use of application packagers and installers such as Microsoft Systems Management Server.

With the advent of cloud computing the requirements for consistency are still present and just as relevant. However the methods and tools used to create cloud infrastructures are now much more akin to application code than the shell script / batch job methods of the past (although some of those skills are still needed). The skills needed to support this are really a mix of both development and sys-ops and have led to the creation of Dev-Ops as a role in its own right.

Recently along with one of my colleagues I was asked to carry out some work to create a new AWS based environment for one of our customers. The requirements for the environment were that it needed to be:

• Consistent
• Repeatable and quick to provision
• Scalable (the same base architecture needed to be used for development, test and production just with differing numbers of server instances)
• Running Centos 6.3
• Running Fuse ESB and MySQL

To create the environment we decided to use a combination of AWS CloudFormation to provision the infrastructure and Opscode Chef to carry out the installation of application software, I focussed primarily on the CloudFormation templates while my colleague pulled together the required Chef recipes.

Fortunately we had recently had a CloudFormation training day delivered by our AWS Partner Solutions Architect so I wasn’t entering the creation of the scripts cold, as at first the JSON syntax and number of things you can do with CloudFormation can be a little daunting.

To help with script creation and understanding I would recommend the following:

• A decent text editor with JSON support – I use Notepad++ combined with the JSON Viewer Plug-In
• A copy of the AWS CloudFormation documentation
• Have a look at the AWS sample templates
• Document the template requirements (i.e. what is it the infrastructure that template needs to create)

For the environment we were creating the infrastructure requirements were:

• VPC based
• 5 subnets
  • Public Web – To hold web server instances
  • Public Secure – To hold bastion instances for admin access
  • Public Access – To hold any NAT instances needed for private subnets
  • Private App – To hold application instances
  • Private Data – To hold database instances
  • ELB
    • External – Web server balancing
    • Internal – Application server balancing
    • Security
      • Port restrictions between all subnets (i.e. public secure can only see SSH on app servers)

To provision this I decided that rather than one large CloudFormation template I would split the environment into a number of smaller templates:

• VPC Template – This created the VPC, Subnets, NAT and Bastion instances
• Security Template – This created the Security Groups between the subnets
• Instance Templates – These created the required instance types and numbers in each subnet

This then allowed us to swap out different Instance Templates depending on the environment we were creating for (i.e development could have single instances in each subnet whereas Test could have ELB balanced pairs or production could use features such as auto-scaling).

I won’t go into the details of the VPC and Security Templates here, suffice it to say that with the multiple template approach the outputs from the creation of one stack were used as the inputs to the next.

For the Instance Templates the requirement was that the instances would be running Centos 6.3 and that we would use Chef to deploy the required application components onto them. When I started looking in to how we would set the instances up do this I found that the examples available for Centos and CloudFormation were extremely limited compared to Ubuntu or Windows. As this is the case I would recommend working from a combination of the Opscode guide to installing Chef on Centos and AWS’s documentation on Integrating AWS with Opscode Chef.

Along the way to producing the finished script there were a number of lessons which I will share with you to help with your installation, the first of these was the need to use a Centos.org AMI from the AWS Marketplace. After identifying the required AMI I tried running up a test template to see what happens before signing up for it in the Marketplace, in CloudFormation this failed with an error of ‘AccessDenied. User doesn’t have permission to call ec2::RunInstances’ which was slightly misleading. Once I’d signed our account up for the AMI then this was cured.

The next problem I encountered was really one of my own making / understanding. When looking at AMIs to use I made sure that we had picked one that was Cloud-Init enabled, in my simplistic view I thought that this meant that commands such as cfn-init that are used within CloudFormation to carry out CloudFormation specific tasks would already be present. This wasn’t the case as the cfn- commands are part of a separate bootstrap installer that needs to be included in the UserData Section of the template (see below):

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
 "#!/bin/bash -v\n",
 "function error_exit\n",
 "{\n",
 " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "ResFuseClientWaitHandle" }, "'\n",
 " exit 1\n",
 "}\n",<br /> "# Install the CloudFormation tools and call init\n",
 "# Note do not remove this bit\n",<br /> "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
 "cfn-init --region ", { "Ref" : "AWS::Region" },
 " -s ", { "Ref" : "AWS::StackName" }, " -r ResInstanceFuse ",
 " --access-key ", { "Ref" : "ResAccessKey" },
 " --secret-key ", { "Fn::GetAtt" : [ "ResAccessKey", "SecretAccessKey" ]},
 " -c set1",
 " || error_exit 'Failed to run cfn-init'\n",
 "# End of CloudFormation Install and init\n", 
 "# Make the Chef log folder\n",
 "mkdir /etc/chef/logs\n",
 "# Try starting the Chef client\n",
 "chef-client -j /etc/chef/roles.json --logfile /etc/chef/logs/chef.log &gt; /tmp/initialize_chef_client.log 2&gt;&amp;1 || error_exit 'Failed to initialise chef client' \n",
 "# Signal success\n",
 "cfn-signal -e $? -r 'Fuse Server configuration' '", { "Ref" : "ResFuseClientWaitHandle" }, "'\n"
]]}}

As the cfn-signal which comes as part of the bootstrap installer is used for messaging to any wait handlers defined in the template this can lead to long breaks at the coffee machine before any feedback is received if they are not present.

The final lesson was how to deploy the Chef Client and configuration to the instances. Chef is a rubygems package, so needs this and supporting packages present on the instance before it can be installed. Within CloudFormation packages can be installed via the use of the packages configuration sections of AWS::CloudFormation::Init which for Linux supports rpm, yum and rubygems installers. Unfortunately for the AMI we chose to use the available repositories didn’t contain all packages necessary for our build, to get around this I had to rpm on the rbel repository definitions before using a combination of yum and rubygems to install Chef:

"packages" : {
 "rpm" : {
 "rbel" : "http://rbel.frameos.org/rbel6"
 },
 "yum" : {
 "ruby" : [],
 "ruby-devel" : [],
 "ruby-ri" : [],
 "ruby-rdoc" : [],
 "gcc" : [],
 "gcc-c++" : [],
 "automake" : [],
 "autoconf" : [],
 "make" : [],
 "curl" : [],
 "dmidecode" : [],
 "rubygems" : []
 },
 "rubygems" : {
 "chef" : [] 
 }
}

Once Chef was installed the next job was to create the Chef configuration files and validation key on the instance. This was carried out using the “files” options within AWS::CloudFormation::Init:

"files" : {
 "/etc/chef/client.rb" : 
 "content" : { "Fn::Join" : ["", [
 "log_level :info", "\n", "log_location STDOUT", "\n",
 "chef_server_url '", { "Ref" : "ParChefServerUrl" }, "'", "\n",
 "validation_key \"/etc/chef/chef-validator.pem\n",
 "validation_client_name '", { "Ref" : "ParChefValidatorName" }, "'", "\n"
 ]]}, 
 "mode" : "000644",
 "owner" : "root",
 "group" : "root"
 },
 "/etc/chef/roles.json" : {
 "content" : { 
 "run_list" : [ "role[esb]" ]
 },
 "mode" : "000644",
 "owner" : "root",
 "group" : "root"
 },
 "/etc/chef/chef-validator.pem" : {
 "source" : { "Fn::Join" : ["", [{ "Ref" : "ParChefKeyBucket" }, { "Ref" : "ParChefValidatorName" }, ".pem"]]},
 "mode" : "000644",
 "owner" : "root",
 "group" : "root",
 "authentication" : "S3Access"
 }
}

The hardest part of this was the validation key, as we had multiple instances wanting to use the same key we decided to place this within an S3 bucket and pull the key down. During the script creation I tried multiple ways of doing this, such as using S3Cmd (which needed another repository and set of configuration to run) but found that using the files section worked best.

Once Chef was installed the client was started via the UserData section (basically a shell script), this then handed control of what additional software and configuration is installed on the instance to the Chef Master. How much Chef does at this stage is a bit of a balancing act as the wait handler within the template will fail the stack creation if its timeout period is exceeded.

As you can probably tell if you have got this far, the creation of the templates took quite a few iterations to get right as I learnt more about CloudFormation. When debugging what is going on it is worth remembering that you should always set the stack to not rollback on failure. This then allows you to access the instances created to find out where they got to within the install, as the UserData section is basically a shell script with some CloudFormation hooks, more times than not the faults are likely to be the same as you would see on a standard non-AWS Linux install. Also for a Centos install remember that the contents of /var/log are your friend as both cloud-init and cfn-init create log files here for debugging purposes.

After watching Werner Vogels keynote speech from AWS Re:Invent it’s clear that treating infrastructure as a programmable resource (i.e. using technologies such as CloudFormation and Chef) is somewhere organisations need to be moving towards and based on my experience so far I will be recommending using this approach on all future AWS environments we get involved with, even the small ones.

Whilst there is a bit of a learning curve the benefits of repeatable builds, known configuration and the ability to source control infrastructure far outweigh any shortcomings, such as granular template validation which I’m sure will come with time.

If you have any comments or want to know more please get in touch.

 

If the world is going Big Data crazy then you need to know what it is, what it isn’t and what it means to your enterprise

The subcategory called Big Data is emerging out of the shadows and into the mainstream.

From left: Matt Wood, Chief Data Scientist at Amazon Web Services (AWS) with Robin Meehan, CTO at Smart421
Photo by Jim Templeton-Cross

What it is.

Definitions abound (who would have thought it? – quite usual in the technology market). For Big Data, we quite like the definition that originated with Doug Laney (@doug_laney), formerly META Group, now a Gartner analyst. It goes something like this:

 ” … increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources)”

Gartner continue to use this “3Vs” model for describing Big Data.

Unsurprisingly, others are claiming Gartner’s construct for Big Data (see Doug’s blog post, 14 Jan 2012).

Still confused?

Put another way, Big Data is commonly understood to be:

“… a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage,search, sharing, analysis,and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.” read more on Wikipedia.

Big Data could be executed on-premise if you have sufficient compute and storage in your corporate data centre. And some do, especially some large banks, and with good success. Several solutions are already out there on the market;  Oracle’s Big Data Appliance is just one example.  But it does also beg the question “why would you” ?

If you don’t want the CapEx of purchasing more tin, or don’t want to gobble up capacity in your own data centre, then there are alternatives. For example, a cost model now exists with cloud-based compute and cloud-based storage (for example, think of Amazon’s announcement of 25 percent reductions in the price of Amazon S3, it’s storage solution) that puts Big Data in the Cloud well within the reach of all UK enterprises. A cost model like that islikely to win friends in procurement and in corporate governance as well as in IT.

Hinging on technologies including Apache Hadoop clusters, Amazon Elastic Map Reduce (Amazon EMR) and others, Big Data is delivering a degree of analytics and visualisation not previously possible at affordable levels.

Don’t just take our word for it, ask around. We could point you to other experts in Big Data, such Matt Wood ( @mza ), Chief Data Scientist at AWS.

What it isn’t.

Big Data isn’t business intelligence (BI). What I mean is that Big Data isn’t BI in any traditional sense of the term. It is altogether another level on from that. Granted that some tooling enterprises may own may be recycled for use in Big Data analytics. But it isn’t another species, it’s another race.

Big Data isn’t a lame attempt at reviving a management information system (MIS); those should be left to rest in peace.

What it means for you.

By now, if you’ve read this far, something should be niggling away at you that you could be missing a trick. I trust it won’t be those voices in your head again. But it might be your instincts telling you how Big Data could answer those tough business questions – y’know, those “I can’t be asked” questions that existing systems just cannot deliver.

Now, you would not necessarily get our CTO to come right out and say that Big Data is the next big thing. But evidence we are assembling so far does seem to point to a new capability to deliver. For those with an appetite to understand their business in new ways, Big Data is delivering tangible intelligence that lets them see new dimensions, new possibilities and new revenue streams.

I did get a full radar lock on something our CTO said in the summer. It was a throw away line at the time but it stuck with me and with others. So, when the time came to consider an appropriate go-to-market message for our quarter three (Q3) focus, we decided to wheel out his one-liner as part of our messaging.

“It’s not about survival of the fittest -
it’s about survival of the best informed”
Robin Meehan, CTO, Smart421 Ltd.

Making no apologies to Charles Darwin or evolutionists, the statement is resonating with decision makers in the enterprise space, not least those in the Insurance sector. Why?  Well, we think it is because a lot of the big insurers operate under many names in their brand portfolios.

The capability to see and understand impacts of brand activities, such as Insurance Quotes, delivered using Big Data analytics in the AWS Cloud, is illuminating new gains that would otherwise have remained out of reach.

Don’t forget – brand analysis is only one use case for Big Data in the Cloud.

If the world is going Big Data crazy then you need to know what it is, what it isn’t and what it means to your enterprise.

Agree?  Disagree?

UPDATE 05 Dec 2012 – Our economist friend Tim Harford  (@TimHarford) sent this hilarious tweet: The data do not lie. OR DO THEY? Muah huah huah! http://dlvr.it/2b2NS1

UPDATE 06 Dec 2012 – Robin and colleague Ben Baumguertel (@bbaumguertel) are attending the Big Data Analytics event in London today (organised by @WhitehallMedia ).

Please Rate this post and Like this post below. If you can, please leave a Comment.

 

The itSMF Conference 2011

Along with Andy and Adam, I attended the annual IT Service Management Forum (itSMF) Conference in London on the 7^th and 8^th November 2011. This is regularly billed as the event of the year for service management professionals and everything ITIL.  It is supported not only by a range of delegates with serious interest in Service Management but also governing bodies, sponsors and 40 suppliers exhibiting their Service Management based tools, training and consultancy.  Of note, that may well be reflective of these times of austerity, is that the last time I attended the itSMF conference in 2009 there were over double the number of suppliers exhibiting at the conference with some notable leaders in the field not represented at this year’s event.

The main content of the conference were 50 presentations on a variety of subjects spread across the 2 days.  Typically the presentations ranged from thinly disguised vendor sales pitches, through theoretical rather than pragmatically based concepts, to real world examples of initiatives undertaken and the associated experiences. As a general rule of thumb it tends to be these real world experience based presentations along with targeted factual update sessions that provide the most value.

The sessions I went to varied greatly in their interest and usefulness with the main value coming from sessions covering:-

• Developing Effective Performance Management, which although not a particularly revolutionary concept did help crystallise a few ideas regards a more system based approach for Performance Management.  Such an approach would be more holistic but provide differing levels of granularity in order that ‘default’ levels provided could be matched to respective audience requirements, whilst at the same time allow for drilling down to more granular based information as and when required.  One interesting idea was to utilise the Service Catalogue to identify the relative value of the Services offered to the business.

• The ITIL 2011 Update session which outlined the major changes that have been introduced in the very recent publication of updated versions of the five ITIL manuals.  Whilst it is clear that we will need to reference all these new updated versions, approximately 80% of the changes are focused on the Service Strategy manual and cascading of these changes through the other manuals for consistency.  The only potential new concept is the introduction of a Continual Service Improvement Register, a concept we have advised clients on previously even if not using that specific terminology.

• A session describing the currently under development ITIL Master Qualification. This will be the highest qualification attainable in ITIL and will be based upon validation of  the capability of candidates to apply the principles of ITIL in the real world. It will not be based upon any form of courses or examinations but instead will take the form of four stages working through Application, Proposal, Work Package creation and submission based upon real world utilisation and experiences before a Final Interview. There will be a set of 40-45 requirements that will be have to be met for candidates to be accepted onto the qualification programme, but further details are not yet available. Therefore the applicability to representatives from Smart421 is not yet known, but besides the personal value of such a high qualification there may be of value to Smart421 as a significant differentiator for Sales if anyone within Smart421 were to achieve this qualification.

• Mobility, Big Data and Precognitive Support was a session presented by Chris Dancy,  the founder of ServiceSphere, which delivered an interesting and compelling view of the recent past, the present and what we should anticipate for the future. For example:-
  • Mobility:- There are 5.3 billion mobile subscribers (that’s 76 percent of the world population) with the growth being led by China and India.  Over 85 percent of new handsets are Web Enabled, which will facilitate continents like Africa to completely leap-frog  ‘hard’ infrastructure requirements like land lines and PC based networking. Mobile devices are already ubiquitous and will be the de facto integration method i.e. will increasingly take precedence and preference over ‘traditional’ PCs and laptops etc.  Facilitating integration will be primarily based around mobile apps,  one projection being that by 2016 cumulative mobile apps downloads will reach 44 billion.
  • Big Data:– a term the presenter thought will become as familiar as Cloud in the coming months and years.  Statistical examples included that the same amount of worldwide information generation that occurred between 1995 and 2000, was achieved in 2 days in 2010 and in 1 day in 2011. Another example was that the size of the internet in 2003 is the same size as Facebook alone  is today!
  • This amount of data is facilitating a move towards more precognition analysis etc. For example:- studying moods on twitter via automated textual analysis, happiness on foursquare mapped across cities etc. In addition to the presentation but another directly connected example, I heard a radio interview on the way home from the conference about perception of value for money of 3D films based upon automated textual analysis of blogs on film based sites. Back in the presentation these techniques are already apparently being utilised in extended news cycles by media organisations, with pre-event analysis occurring specifically in preparation for actual events – sounds a bit like pro-active trending!
  • Finally the speed of change that should be anticipated is already outlined to a large degree with some of the statistics highlighted above, but it is also argued that in the wider scale that “we are still in the first minutes of the first day of the internet revolution” and that Kurzweil’s Law of Accelerating Returns with the implied exponential and non-linear growth in technological capabilities and human knowledge is already occurring in line with Kurzweil’s 6 Epochs  ideas. Whilst it could be argued that such considerations are overly cerebral and abstract, it doesn’t take much consideration of the evidence and experience of the last 10 or 20 years to realise that the future opportunities and challenges are going to be a world apart from what we have seen previously.

Finally Andy & I also attended a couple of sessions based on management of ‘The Cloud’. We found both sessions basic and were left with the clear impression that Smart421’s capabilities and position are far more advanced than anything represented at the conference – so clearly this is an area we should be shouting about more!!

 

Enterprise Architecture and BPM Conference 2011

Recently, I attended the 3-day conference in London that combined, for the first time, EA and BPM, which had in previous years been the subject of separate conferences, see the overview at http://www.irmuk.co.uk/eac2011/overview.cfm for more details. Thanks to Robin Meehan presenting a session with Visa Europe we got a good deal on the ticket to go to all three days including the Seminar on Wednesday.

The first day gave me the opportunity to see the legendary John Zachman present a half-day introduction to his famous “Zachman EA Framework”. The seminar was subtitled “Enterprise Physics”, which made me think of Star Trek and Scottie the Engineer but maybe that’s just me. Zachman prefers using the terms “ontology” or “classification” rather than “framework” for the core 6×6 matrix (sorry, “normalised schema”) that compares with the periodic table that underlies the whole of chemistry. The main thrust of Zachman’s very entertaining presentation was that nobody can carry out any seriously complex activity without architecture and that architecture is the same for enterprises as it is for aeroplanes or one hundred storey buildings (but harder).

The analogies and application of EA to science and engineering showed how relatively young and immature is the whole practice of EA and Zachman can rightly claim to be a pioneer in the late 1960s and still going strong now at the age of 76. Robin Meehan wrote about him two years ago and I would echo a lot of the sentiments he expressed regarding the energy and passion he still displays.

In the afternoon on day one I attended a seminar on Business Process and BPMN, which told me that BPMN 2.0 has only four basic building blocks that result in 100 or so detailed objects with embellishments and decorations. For example there are something like 63 different categories of “event”. What BPMN 2.0 does is categorise into “common core” of just a few important fundamental concepts that can code the majority of simple business processes. There were a range of tool vendors in the exhibition supporting BPMN in various ways, many now based on standard archimate-style notation.

What surprises me a little bit is the way the business process delegates still seem to think they exist alongside EA whereas by definition EA encompasses the whole enterprise, as Zachman says “The whole thing including the business architecture and processes”, so therefore BPM falls within EA.

Day Two

Day 2 started with a nice opening by Sally Bean (@cybersal on twitter – Twitter was in evidence including tags #IRMEAC and #IRMBPM that I used for a bit) and Roger Burlton (@RogerBurlton) that focused on having a disciplined, coherent and shared architecture strategy that encompasses both EA and BPM; ok, I would argue EA already encompasses BPM but it’s good the similarities and overlaps are now being recognised and acted upon. The other statement that stuck with me was that “The common repository” is critical, something that causes a debate in our group with respect to federated SOA and autonomy of business units within an enterprise.

The keynote was given by Thomas Lawton (@TCLawton) who was clearly suffering with mild laryngitis so has to be applauded for getting through his description of breakout strategy, leadership and vision wheels so well. Some nice categorisations of businesses in frame of their response to the recession (Panic, Protect, Cloak, Conquer) and then in terms of breakout, being offensive (in the “attacking” sense in British parlance), i.e. “…the best form of defense is attack”. He spent a long time exploring the nature of growth opportunities, where Google are a “True Original” taking an emergent market by storm and Tesco are a “Big Improver” moving from laggard to leader in established market. I stopped to think about it and would probably categorise Smart421 as “Wave Rider”, not really a true original but taking on and leading the way in an emerging market (EA Consultancy). The only thing that bothered me slightly was the example in this space was Ryan Air – I’d like to think we have a much friendlier customer focus! Thomas’s “Vision Wheel” was an interesting concept, separating external and internal aspects and the final section was about how to create a “Magnet company” that excites markets and attracts customers. The key seems simply to build the Vision for the future based on the six aspects: Price, Features, Quality, Support, Availability, Reputation. I had a go at doing this for Smart421 below. It would be interesting to get other peoples’ views on the ratings.

The afternoon keynote from Ian Gotts of Nimbus focused on CEOs and specifically selling BPM projects to CEOs. The first rule he quoted was “not BPM”, which was a theme of some other talks “Don’t mention architecture”. It reminded me of the famous football autobiography by Len Shackleton where he entitled a chapter “What the average club chairman knows about football” and left the page completely blank. Gotts’s talk used examples from the transformation of Carphone Warehouse from a basic “phone shifter” to a rounded customer-oriented gadget shop with supporting processes. The slides contained some interesting predictions like the market for BPM services to top $24bn in the next few years and he had a nice graphic showing an exponential increase in spending by 8 of their customers recently (could just be coincidence as business always increases over time). It was entertaining and made me more aware of how to present to senior business-people, as if I didn’t already know not to mention IT terminology.

Also today, I had the pleasure of attending two presentations by working Enterprise Architects from Shell and British Gas. It is always enjoyable hearing about real-world experiences that highlight gaps in the models. Dan Jeavons from Shell is far too youthful to know as much as he does about Enterprise Architecture but I found myself agreeing with what he was saying and it confirms my belief that implementing EA needs sponsorship from the top and there is a right way to do it (meta-model definition before tooling for example).

Jane Chang from British Gas pretty much developed her own practice, on the back of delivering Smart Metering to the company’s 10 million customers. The programme has been a great success and now has a large 400-person development team working on it to meet the architecture vision. A very good end to the day.

Day Three

And so to the third and final day of EAC and BPM and the obvious highlight was the presentation bySmart421 CTO, Robin Meehan and Chris Forlano Lead Enterprise Architect at Visa Europe on “Maturing Visa’s Enterprise Architecture Practice”.

Robin Meehan CTO at Smart421 (pictured left) with Chris Forlano, Lead Enterprise Architect at Visa Europe. Photo by Andrew Smale.

The session was appreciated by all and they asked some very interesting questions, like “How did you justify a 530 days budget for this work?”, which should probably have been answered by Mark had he been there.

Prior to that I went along to a Lean Six Sigma presentation and learnt a few more strings to use around promoting Quality through reducing variance (Six Sigma) at the same time as addressing the 7 Sins of Waste (Lean). I thought Peter Matthijssen was really good at using examples to introduce LSS as a practice for aspiring Business Process Architects and explained the concepts really well.

The morning keynote was probably the best talk of the whole conference by Jason Bloomberg on …. you’ve guessed it… The Cloud!   Or more specifically, “Architecting the Cloud – How EAs should think about Cloud Computing”. Both the Pros and the Cons were presented and the not so subtle message to delegates was to not let vendors drive down the route of private cloud and that public cloud cannot be trusted. I did think some examples: a Cloud employee taking a memory stick to your server and stealing your data, or the police impounding your (shared) boxes because of illegal activity by someone else was a little bit OTT. The main message reinforced our view that you must architect for the cloud and synergies with SOA were well presented, in particular the suggestion to extend SOA Governance to cover Cloud Governance, a reasonable extension as I’ve always thought SOA Governance should govern the underlying platforms for capacity and autonomy anyway. I didn’t quite get his point of Cloud services using REST couldn’t be governed as part of SOA because surely SOA is technology agnostic? His last slide on availability and redundancy with reference to the April Amazon outage provided a good discussion point afterwards and if anything this will be good for service providers like Smart421 offering experienced Cloud consultancy.

My second session of the day was “The Success of a Pragmatic Enterprise Architecture approach ‘STREAM’” by Jaap Schekkerman, Thought Leader Business Technology Strategy. I wasn’t completely convinced that these methods will work for everyone and the recommendation to design business methods on A0 format was provocative to someone like me who believes in a more componentised approach and that a process should fit on a single page to be understandable. Some of his slides also suffered from the A0 format and were incomprehensible. However, I did like Jaap as a presenter and he does have some original methods built into STREAM, which stands for:
Speedy Traceable Result-driven Enterprise Architecture (or Asset/Agile) Management, and it can be integrated with other frameworks and methodologies.

If I have one regret from this conference it is some of the session choices I made – Oliver Robinson’s presentation about improving the National Policing Agency drew a lot of praise, as did Tom Graves from Tetradian on “Respect as an Architectural Issue: a Case Study in Business Survival” but you can’t be everywhere. At least I have all the slides and further references like to tetradianbooks.com for the last one.

I admit I also suffered a little bit of BPM-fatigue after a while of going round the numerous vendors and trying to understand their products. However, if anyone has a need to deliver a BPM tool then I’ve got some good contacts now and a backpack full of literature and demos so give me a call or tweet me @smaley

 

Why does AWS stress security in the Cloud?

Richard Holland, Operations and Delivery Director of Eagle Genomics addresses delegates at the 2011 AWS Tech Summit, London 17 March.

The AWS Technical Summit in London 17 March [see http://aws.amazon.com/aws-tech-summit-london-2011/] was very worthwhile and no one could fail to notice that AWS themselves were taken by surprise in the exceptional attendance levels. Along with colleagues, Smart421 attended as AWS Solution Providers.

One of the key takeaways for me was how AWS continues to be highly responsive to the market. As well as bringing features to market in rapid succession, they have also listened and replied convincingly on the recurrent obstacle of security in the Cloud.

In fact, in addition to AWS’s already comprehensive security stance [see http://aws.amazon.com/security/] it emerged that one of AWS’s customers, Eagle Genomics based in Cambridge UK, had also permitted two independent IT consulting firms, AT&T and Cognizant, to perform ethical hacking on their AWS instances (permissions obviously required in advance).  The outcome?  Both firms reported that it couldn’t be done.

For hardcore sceptics (aren’t we all at heart), AWS say they have a list of partners that provide services around ethical hacking. This means that if your business case warrants a belt and braces approach on security, it’s possible to engage IT consulting firms these kinds of trials dedicated to your instance or instances in the Amazon cloud.

In reality, with some notable exceptions where systems are understandably internalised, both physical security and digital security offered in AWS is far in excess what the majority of organisations are able to provide for themselves. I’m certainly not alone in thinking that a far bigger risk for enterprises resides in the portability of data (e.g. copies held on company laptops, CDs, USB keys, etc) than in hacking instances of Cloud computing, particularly those on AWS.

It would seem that the objection around security in the cloud is being steadily eroded away. About time.

 

“Small opportunities are often the beginning of great enterprises” – Demosthenes

"Small opportunities are often the beginning of great enterprises" - Demosthenes

Did you know that of the 500 original companies of the original S&P 500 (Standard & Poors) in 1957, only 74 of those remained in 1997 – that is a staggering 85% reduction!

Why is this?

The capital markets in the world today encourages rapid and vast growth leading to greater revenues and wealth, meaning people are expecting companies to adapt and perform to the market conditions and are not too understanding when they under-perform over the long term. Of those companies that are no longer in the S&P 500 one of the key factors that they were unable to be keep up with the changing demands of the markets and been able to be agile in entering new high value markets. Whilst change can be slow, it is always powerful!

It is having this ability to adapt and change that helps companies today to survive and a great way to help your business achieve this agility is by adopting a SOA architecture. It can really help a business create an IT infrastructure that will support a truly dynamic enterprise.

SOA is all about defining services that align and support business process, and exposing these services so they can re-used easily. It also allows these services to change independently so only aspects of business processes that need to change (as the business adapts and changes) minimising the impact on the rest of the IT infrastructure and business.

The loose coupling gained from adopting a SOA also provides abstraction from underlying systems and applications. This allows greater flexibility on consuming and invoking the capability exposed through the SOA, whilst enabling the ability to  switch the underlying systems and platforms as and when this is required.

When you consider all of these benefits you can see the agility this provides your business.

SOA is nothing new and has been around for a number of years now (and in a number of different guises) but in our opinion it is now a widely accepted mainstream approach within IT architecture, which we are seeing with a number of our clients investing and driving forwards with SOA as they have realised (and are already reaping) the benefits that it brings and how that journey can be the beginning  of a great and dynamic future.

 

IBM Websphere User Group – 18th March – Bedfont Lakes

It’s been a while since I have attended one of these so I went with an element of excited anticipation…honest

In truth it was good to have a day away from the office and client site to refresh my view on as many Websphere related topics that I could cram into one day. Another important facet of attending the event was to ‘man’ the Smart421 marketing stand in between the various sessions with one of our Lead Consultant colleagues as part of our long-standing commitment and relationship with the WUG. This is always a bit of a daunting prospect but I saw it as a way of ‘dip-sticking’ the current IT temperature gauge – I was kind of expecting some stumbling conversations around the Cloud but there were more conversations around SOA…….a debate for another time but perhaps SOA is still more real for many organisations at the moment as they continue to experience the pain of their SOA journeys. 

The breadth of the sessions available is a real attraction of one of these events and to me adequately justifies any time and cost of attending one these events. We weren’t disappointed by the topics up for discussion. I went to 4 sessions consisting of:

• Business Process Management: Collaborate, Iterate, Refine, Validate – by Waverney Croson, Consulting IT Specialist, IBM
• Java 6 Unleashed: Tuning the IBM JVM – presented by Chris Bailey, IBM Java Technology Center, Hurlsey Lab
• Learning from Other People’s SOA Experiences – presented by John Moe, Head of Integration Services, Tori Global
• Websphere Message Broker V7 Introduction and New Features – presently by Dave Page, IBM Consulting IT Specialist

All the presentations I attended were well prepared and well presented but the one that in the end stood out for me was the ‘Learning from Other People’s SOA experiences’. There has been much debate on the subject of SOA generally and there has been much discussion amongst our consultants in Smart421; where it is today, where it is heading and how it relates to the emergence of the Cloud. Anyway back to the presentation……the slides rolled through but there were a few gems that I thought were worth repeating. The presenter clearly had lived and breathed this stuff and had also been burnt by it; his opening gambit was ‘its not easy’!. He stated that many large organisations had good messaging architectures (hub and spoke) and that moving to a SOA architecture (n layer) wasn’t necessarily a natural next step; “just because you can doesn’t mean you should”. Some reasonable advice was to begin your SOA evolution in small chunks; stay with the core capability of the product you are using to help avoid vendor lock-in around some obscure functionality only available in that product. He did say that the products in this area were far more mature than a few years ago but his strong advice was always to use vendor support to help you through the evolution process. He mentioned that the original implementations of most the products in this area were embarrassingly poor and this did remind me of some work with a client some time ago (circa. 2006) on an early enterprise version of one of these products – it would have been overly generous to even suggest this was ready for ‘beta’ trialing. Some of the other challenges he mentioned were around the adoption of service-based development practices and that organisations entering their SOA evolution should initially look to use agile methodologies to deliver services where tight control could be maintained; when the processes were more mature this could go to a waterfall methodology and off shored. The definition he used for determining SOA maturity was one around reuse; a rough figure of 30-40% reuse of services would represent a mature SOA architecture.

An interesting debate was around governance and it was made clear that this was one of the key barriers to success. This developed into an interesting discussion about versioning of services; this has been raised before on a client site and I was expecting a concise and polished answer but there wasn’t one. The answer given was that there wasn’t a tool out there to help with effective versioning of production services and typically the service would be split into 2 separate production services. He justified this by saying that as soon as a service goes live it effectively becomes legacy and there should be an acceptance that it will be difficult to change due to the potential impacts on consuming platforms. This is certainly something I have seen in practice but I wasn’t expecting that to be the norm!

The session that I was eager to attend based on the original order of the day’s events was the Business Process Management (BPM). I see a lot of need within organisations to manage their processes more efficiently and realise the cost savings that just a few key improvements can often bring. The presentation didn’t disappoint but I was left trying to work out how an organisation could effectively adopt such an array of tools in this area. It may be a reflection of some of the clients I have worked with but it seems to me to require such a large amount of transformation within an organisation to perhaps make it too big a shift to bite off in one go. Maybe the answer is in the question in terms of concentrating on implementing the products/tooling (let’s ignore the cost and vendor lock-in issues for the purposes of this discussion) and target a particular problematic or high-value process and just deliver it. If the delivery is successful (and of course it should be because you will have heavily weighted the first implementation in your favour) then get the recognition and buy-in from other areas and rollout the adoption of the tool and the other requisite skills, training etc. across more and more of your organisation. Perhaps to convince me, I need to attend a ‘Learning from Other People’s BPM experiences’ next time

 

Enterprise Architecture – a pragmatist’s view.

Following a long stint at one of our largest clients, mainly working on Service Management transformation and a review of their Enterprise Architecture capability, I moved into the role of Business Design Authority at a smaller, more dynamic client in October last year.

This is a great opportunity for me to use all of the experience I have gained in my 15 year career to help the client develop a more structured approach to understanding and documenting strategy, processes, structures and requirements. The client has clear business objectives including significant growth within current markets, entry to new markets, improving consistency and effectiveness of their processes and doing all of this in a controlled and sustainable manner.

We could have come in and tried to impose a rigid top-down architectural approach. We could have focused on the lowest level requirements for their new core business system and worked bottom-up. But neither of those would have worked in isolation.

So we’ve taken a middle-out approach. Sort of. What we have actually done is to be pragmatic in helping to make sure that progress is being made against key business milestones and deliverables, whilst bringing in more rigour and structure to the way that things are done. For example…

We’ve confirmed that the strategy, principles and business objectives for the next 5 years are clearly understood and communicated.

We’ve worked with key stakeholders to define and document a new product that will allow the client to enter their key new market segment. We’ve taken this product definition and assessed it for impacts throughout the organisation – what changes to areas such as people, processes, information and technology are needed in order to sell and service the product? And now we’re delving into the detail of each of these areas to ensure that there will be “just enough” capability in place to be able to meet the immediate needs of the business – but in a way that can be scaled quickly as demand for the product increases.

We’re looking at existing processes and have brought in a method and templates for process documentation from “Level 1″ right down to “work instructions” where necessary.

We’ve mapped the entire Application Landscape – a few larger/core systems in a sea of Access databases and Excel spreadsheets. And we’re working on a strategy that will gradually bring more of the “sea” under tighter control, whilst keeping some of the existing flexibility and adaptability. We’re also looking at how and where the applications can be hosted to best meet the needs of the business.

And yes, we’re also keeping a careful eye on the detailed requirements for the new core business system and elsewhere – because, as we all know, the devil is always in the detail.

I could go on. But you get the idea. It’s about doing whatever adds value NOW, but with an eye on a more structured, sustainable, strategic future. It’s about saying what you’ll do and doing it. It’s about keeping the communications channels open so that the whole organisation can find out what’s going on, how this programme of change will help them and how they can help the success of the programme.

And it’s about having some fun along the way!