Consultancy


ForgeRock OpenAM version 11.0 highlights Powerpoint presention slide on 15 October 2013 by John Barco, VP Product Management at ForgeRock. Photo: Kirsten Hurley

ForgeRock OpenAM version 11.0 highlights Powerpoint presention slide on 15 October 2013 by John Barco, VP Product Management at ForgeRock.
Photo: Kirsten Hurley

Last week I had the pleasure of attending the ForgeRock Open Identity Stack Summit: Europe in a chateau on the outskirts of Paris. The event was a sell out but, keen not to turn away delegates, the ForgeRock team valiantly sacrificed their rooms to the extra attendees while they stayed in a hotel half an hour away!

There was a real mix of attendees, from the technical community to procurement executives, and of course ForgeRock’s partner community. All were eager to hear product updates, and listen to industry experts such as Eve Maler (Principal Analyst at Forrester) on the trends they are witnessing across the world. Eve’s key note on modern IAM and trends certainly caused a stir with her statement that XACML (eXtensible Access Control Markup Language) is now in clear decline, and that authorisation standards need to get finer still. UMA (User-Managed Access) is the one to watch apparently…

A lot of the messaging revolved around how IAM (Identity Access Management) is moving to ‘IRM’ (Identity Relationship Management). This is largely driven by factors such as internet scale over enterprise scale –  an inescapable requirement for a user base that is no longer restricted to employees, but partners and customers too, and accessing not just from on premise, but from the internet and in the cloud. And that’s without even mentioning the number of devices each individual is expecting to be able to access your systems with!

It was also apparent why ForgeRock had taken the radical step to rebrand the latest upgrade from v10.2 to v11 when the new features were revealed (see photo below). ForgeRock has so rapidly developed an already market-leading product to become even simpler to integrate and deploy that the changes certainly justified the leap in nomenclature.

Finally, I cannot sign off without mentioning CEO Mike Ellis’s big announcement of the event – Salesforce and ForgeRock announced a partnership which will see the SaaS vendor integrate the open source software into its Identity Connect product.

If there is anyone out there who still wonders whether open source technology really has a place in the enterprise, surely this news that one of the world’s largest technology vendors sees fit to partner with them must mean that ForgeRock’s position in the IAM (or IRM!) market is confirmed?!

Please share this blog using the social icons, or via the short URL http://bit.ly/1aFumKp

Please Rate and Like this blog.  Our readers want to see YOUR opinion, so please take a moment to leave a Comment.

I have just got round to catching up with some blog posts after holiday season and this one from my KCom colleague Rob Wells caught my eye. I particularly liked the word “phablet” to cover phones and tablets but I’m not sure it will catch on.

Rob makes a compelling case against a proliferation of Apps and for building ‘mobile optimised HTML’ which relies on an initial web server detection mechanism that checks the user agent (Rob says Operating System but I don’t think that’s what he really meant) and then directs the request to the right pages. This is absolutely the right approach given that more and more users are now finding your company websites on mobile than on full-scale laptop screens so immediately directing from “www.acme.co” to “m.acme.co” that is optimised for mobile should be a no-brainer.

I like the KCom portal approach of assuming controls are ‘touch-first’ as well. I overheard a conversation the other day where someone was wondering how long it would be until putting something on a web page like “CLICK” (PC or mouse-oriented) would die out in favour of “TOUCH” here. Perhaps somebody needs to invent a new word that covers both options. I tried out an online thesaurus for touch which offered “tap” or “hit” (and a few other slightly dodgy ones) but synonyms for click were even less helpful.

Anyway, I digress. Rob has made a very good argument for HTML(5) on browser being the target of choice but I would argue it is not always the best option.

Here’s just two reasons why:

1. You’re NOT always online. The Apps I use the most are the ones where I can use them offline (and occasionally synchronise data when the Interweb is available) for example EverNote, Strava or good old faithful email clients. I actually hate Apps that rely on Internet connection to function and avoid them wherever possible.

2. There isn’t very consistent or reliable support for HTML5 in many browsers yet and it’s just as hard to ensure your application works on web browsers as it is with native or “hybrid” Apps with lots of browser-specific catches that you need to work with. For the foreseeable future I can’t see pure web applications having enough functionality to compete with Apps which run natively on the popular handsets.

The nice thing about Hybrid platforms like IBM Worklight or Appcelerator is that you can still use a lot of the common HTML, CSS and JavaScript skills but write them to use APIs into the native features such as encrypted cache through JSONStore for offline storage and benefit from a little bit the best of both worlds. Worklight, which is now a leader in Gartner’s Magic Quadrant of Mobile Application development Platforms has the option to generate a mobile web version as well as version for iOS, Android, Windows 8 and BlackBerry so you can have your cake and eat it. It also addresses the problems highlighted in Rob’s blog of fragmentation of the different mobile platforms plus controlling versions and ensuring security.

There are advocates of both approaches and neither is really right or wrong – as always you need to take a decision based on your requirements, strategy and budget.

Today myself and four other Smarties attended Norfolk’s first Mobile Development Conference at the Hethel Engineering Centre, which is right next to where they make Lotus Cars.

Conference Room

There is an obvious tie-up between Hethel and Lotus given that the main presentations were held in the Colin Chapman room (founder of Lotus cars) where one of Ayrton Senna’s “99T” F1 cars was stuck to the wall!

Mobile Development is one of the most exciting and diverse areas in IT at the moment and this conference did very well to have a wide coverage from games developers like MonoGame to Tim Ferguson, Head of Digital at one of our customers AVIVA and their mobile app lessons learnt from their various innovations and experiments.

The keynote by Neil Garner of @Proxama resonated with me very much, both in his memories of tech from past years (Nokia 7110 first Wap phone) to his honest assessment of NFC and rebuttal of the doubters who don’t see NFC taking off now. The ARM Trustzone was highlighted by Neil as a key element in providing security for NFC applications. There are Contactless terminals everywhere now and 9 of the top 10 device manufacturers are signed up to support NFC – Apple is the odd one out but aren’t they always?

Our own @JamesElsey1986 later showed that NFC is more flexible and powerful than you think using Android. James later tweeted:

Source code/slides from my #NFC workshop http://ow.ly/mDz7A  Feel free to ask questions / give feedback. Thanks for attending! #MobDevCon

Matt Lacey presented two sessions, his first on tips for developing for Windows 8 included some real gems which will help us with our tailoring of our cross-platform Apps to work well on the new Windows platforms. I agree with Matt, who worked on PhoneGap’s Windows integration code that you have to be knowledgeable and experienced in developing native Apps to be able to build successful cross-platform Apps. Luckily Smart421 have a whole Microsoft practice to help us Java-oriented types out with that. Read Matt’s blog for more info and his slides from his second presentation on monetising Apps.

I was first on to present after lunch and talked about our work delivering cross-platform mobile experiences with Worklight – my slides are now up on slideshare. There was a general theme at the conference that cross-platform tools are coming of age and the compromise of user experience and performance when compared to native development is far outweighed by the much faster and cheaper overall costs of App development and maintenance. I just about managed to demo the new Worklight 6 Studio IDE and Console. I am really liking the improved jQueryMobile integration and want to find time to check out the new App Centre tools and automated testing when I get the chance.

Ruth John (@rumyra) of O2’s “The Lab” gave a kitty and puppy-tastic presentation on FireFoxOS and why Telefonica have taken it up especially in the emerging South American markets – it’s free, works well on low-end handsets with the FireFox operating system built on top of the Gecko layer as is Android. It will be really interesting to see if this will catch on in the UK and European markets in these times of austerity where people are perhaps not quite ready to splash a few hundred every year on the latest iOS gadgets.

There was also a really enlightening “sponsor presentation” by Basho on the subject of reclaiming the terms web scale, big data, dev ops and how the NHS is using Riak’s open source technology.

Massive thanks to Naked Element (Paul and Marie) and everyone involved in setting up the event, thanks to Hethel for such a great venue, the sponsors for the delicious lunch and the attendees for their support and kind comments.

P.S. Welcome to twitter @CharlesBSimms :-)

Photo © Ciprian.d Stock Free Images & Dreamstime Stock Photos

Photo © Ciprian.d Stock Free Images & Dreamstime Stock Photos

As you already know from the blog 5 Dec 2012, as we were preparing to tell  the world about our Big Data capabilities, our CTO Robin Meehan coined a cracking one-liner:

“It’s not about survival of the fittest – it’s about survival of the best informed”

We liked it so much we decided to include it on our go-to-market materials.

Since then, we’ve witnessed some upsides such as how our customers have decided to deliberately ‘think outside of the box’ to better understand their portfolio of brands and be better equipped to attract customers without canabalisation of other lines of business.

A great case in point would have to be Quotemehappy.com which is already taking advantage of the analytical power in cloud computing with Big Data on AWS to understand its brand and build its business.

Aviva launched Quotemehappy.com in August 2011. With a national reputation and strong brand in the general insurance sector at stake, they wanted to know how their multi-touchpoint cross marketing activities impacted each brand in their enterprise portfolio.

Smart421 was invited to assist.  Smart421 architected the Cloud instances using Amazon Web Services (AWS) and developed the algorithms needed to maximise the power of the customer’s own data and the Big Data analytical environment. This gave the business a level of insight not previously possible with traditional on-premise business intelligence tools and techniques.

And the customer was kind enough to go on record about what we had been able to do.

“Smart421’s Cloud architects gave us a head start on making Big Data real for us, including how business insights are really delivered, what the costs really are, and how the technology really works in our context. Their output contributes to how we differentiate ourselves in a crowded market.”
Keith Misson, Operations Director at Quotemehappy.com (an Aviva company).
Naturally, we asked if we could feature this on our website because it evidences the transformation effect of what a well architected IT strategy can deliver for a business.

(Why not share this case study with your colleagues and friends using short URL http://bit.ly/Wby5Y6 )

Big Data is a good example of how technologies developed for one use have been deployed for an altogether different use. I think that Robin’s original quip on survival has actually gone on to deliver a powerful lesson on technological exaptation.

 

Please Rate and Like this post, If you can, please leave a Comment.

Turning disc

Turning disc by Toshiyuki IMAI.
CreativeCommons

With most enterprise arrays now providing the ability to allocate more storage than is physically installed and with data moving transparently between tiers of low cost and high performance with no human intervention – what could possibly go wrong?

For a long time now SAN infrastructure has been at the core of the Data Centre satisfying a need to hoard as much data as could possibly be collected. With the required capacity growing at an exponential rate, it has become common to predict massive data growth when designing services, often calculating growth for two or more years. This scenario leads to large amounts of storage being purchase in anticipation of things to come, by the time two years has passed, some of the attached systems will become legacy as they are superseded by shiny new solutions and some of the systems will just not have grown as expected. Even those systems that have behaved as designed are probably using no more than 70% of the storage as likely as not due to a “fudge factor” arbitrarily thrown into the mix by the architect.

Virtualised storage arrays (each in it’s own way it would seem) allow us, within reason, to allocate as many devices of whatever size we like to as many hosts as we see fit with only a small pool of physical disk. Allowing us to satisfy our storage hungry analysts whilst buying the disk in a phased manner and delaying spend until it is necessary. Add into this the reducing cost per Gigabyte over time and the benefits mount up.

At the same time, the array architecture has developed in such a way that it is harder to facilitate small amounts of physical growth. With larger RAID groups becoming more common and larger disk sizes compounding the issue, traditional SAN allocations became inflexible and for a small requirement, potentially expensive. The smallest building block becomes a RAID Group and for a one Terabyte requirement on a traditional storage array might require the business to grow an array by thirty Terabytes or more. With virtualisation comes the ability to create more devices for allocation without having to have physical storage installed to match. There is no longer a need to grow a system until the written data exceeds thresholds and the larger building blocks are far more palatable when they are shared between all attached systems.

As with everything in life, there is always a flip side. The balance in this case is increased risk; what if the physical pool fills up. At the very least, writes will have to stop but in some cases I/O stops altogether, neither scenario is one that a business will want to contemplate. This is where the need for careful planning and monitoring is essential.

In a simple virtualised environment, physical storage will be configured into Storage Pools and a number of virtual devices created and attached to these pools. As data is written to the logical device it will be written across the physical disks making up the storage pool.

Considerations when designing virtual solutions:

  • The number of servers that will be connected – To provision for sufficient connectivity and I/O capability. It is often not available but any additional I/O profiling for attached systems will also be useful.
  • The average server allocation size and utilisation – To calculate required space for allocation and physical writes.
  • The maximum allocation size –  Large allocations could result in a single server having the ability to fill all available pool space.
  • Maturity of the service – Mature systems will require larger initial space with slower growth; new systems may start small and grow into the allocated space over an extended period.
  • Performance – Is there sufficient throughput for the attached servers?
  • Criticality – There is an associated risk with over provisioning. There are ways to mitigate this and in some cases systems will be too critical to the business to accept any additional risk.
  • Mixed Production, Test and development environments – Will the different environments share the same pools or should they be ring fenced?
  • Alerting  – What level of alerting is configured? At what level of utilisation do warnings start? Can unexpected write bursts from a single application be identified and highlighted?
  • Capacity for growth – Can the pool be quickly expanded?
  • Time to deploy – How quickly can the vendor react to increase the Pool size.
  • Plan B – Can space be redeployed to production systems?
  • Cost reduction per GB – Flexibility may be cited as a benefit but the main driver for virtualised storage is to drive the cost per GB down, on the most part this is achieved by deferred purchase and tiered storage.

Monitoring:

With thin provisioning, monitoring is essential to maintaining a low risk environment.

As a minimum the capacities must be monitored on a regular basis depending on the activity in the environment, the attitude to risk and the subscription and utilisation figures. In most large environments daily should be sufficient or even weekly.

We have found the following capacity metrics to be the most useful and collect on a weekday basis so that we can produce trend analysis to support future forecasts.

For each pool on each array we collect:

  • Installed storage
  • Used
  • Virtual TB
  • Allocated
  • Subscription
  • Nett subscription

Experience shows that the larger environments have a very linear growth both in allocated and written storage and the trending figures provide quite accurate forecast estimates.

Performance figures should also be reviewed but a less regular basis is sufficient – weekly or even monthly.

With virtualisation improving the overall utilisation and ever larger physical disks being installed we see as a result that the I/O profile per disk is also increasing. These performance trends should be monitored and reviewed to anticipate thresholds being reached.

As a result of the increased utilisation and larger physical disks, we are also seeing the number of host connections to front-end ports (fan-out ratio) and the amount of storage allocated to front-end ports both increasing. The host ports should be monitored in the same way as the disks to anticipate performance thresholds being reached.

There is no point increasing the size of the pools to accommodate further allocations if the path to the data is already fully loaded.

This performance monitoring is specific to the capacity planning and does not replace the normal daily performance monitoring and alerting on the storage arrays.

What can be done to prepare for emergencies?

Most virtualised storage solutions have the functionality to preallocate storage. For systems that cannot afford any outage the storage can be preallocated in advance.

Understand the process with the vendors to increase the installed physical capacity.

  • How long for the vendor to install new storage from the request?
  • Can elements (such as procurement) be prepared in advance of the need or post deployment to avoid delaying the process?
  • Can hardware be left on site to reduce hardware deployment timescales?
  • Are sufficient resources available to accept additional disk?
    • Free drive bays
    • Cache
    • Power
    • Floor space

Understand the environment.

  • What systems are attached that could be sacrificed to maintain critical services?
  • Are there device replicas in the pool for non-production purposes that could be released?

These observations are based on simple virtualised pools, most hold true for tiered storage environments but these also bring their own sets of concerns and even more metrics to follow.

Please Rate and Like this blog. If you can, please leave a Comment.

ImageIf you have any experience of supporting large scale infrastructures, whether they are based on ‘old school’ tin and wires, virtual machines or cloud based technologies you will know that it is important to be able to create consistently repeatable platform builds. This includes ensuring that the network infrastructure, ‘server hardware’, operating systems and applications are installed and configured the same way each time.

Historically this would have been achieved via the use the same hardware, scripted operating system installs and in the Windows application world of my past the use of application packagers and installers such as Microsoft Systems Management Server.

With the advent of cloud computing the requirements for consistency are still present and just as relevant. However the methods and tools used to create cloud infrastructures are now much more akin to application code than the shell script / batch job methods of the past (although some of those skills are still needed). The skills needed to support this are really a mix of both development and sys-ops and have led to the creation of Dev-Ops as a role in its own right.

Recently along with one of my colleagues I was asked to carry out some work to create a new AWS based environment for one of our customers. The requirements for the environment were that it needed to be:

  • Consistent
  • Repeatable and quick to provision
  • Scalable (the same base architecture needed to be used for development, test and production just with differing numbers of server instances)
  • Running Centos 6.3
  • Running Fuse ESB and MySQL

To create the environment we decided to use a combination of AWS CloudFormation to provision the infrastructure and Opscode Chef to carry out the installation of application software, I focussed primarily on the CloudFormation templates while my colleague pulled together the required Chef recipes.

Fortunately we had recently had a CloudFormation training day delivered by our AWS Partner Solutions Architect so I wasn’t entering the creation of the scripts cold, as at first the JSON syntax and number of things you can do with CloudFormation can be a little daunting.

To help with script creation and understanding I would recommend the following:

For the environment we were creating the infrastructure requirements were:

  • VPC based
  • 5 subnets
    • Public Web – To hold web server instances
    • Public Secure – To hold bastion instances for admin access
    • Public Access – To hold any NAT instances needed for private subnets
    • Private App – To hold application instances
    • Private Data – To hold database instances
    • ELB
      • External – Web server balancing
      • Internal – Application server balancing
      • Security
        • Port restrictions between all subnets (i.e. public secure can only see SSH on app servers)

To provision this I decided that rather than one large CloudFormation template I would split the environment into a number of smaller templates:

  • VPC Template – This created the VPC, Subnets, NAT and Bastion instances
  • Security Template – This created the Security Groups between the subnets
  • Instance Templates – These created the required instance types and numbers in each subnet

This then allowed us to swap out different Instance Templates depending on the environment we were creating for (i.e development could have single instances in each subnet whereas Test could have ELB balanced pairs or production could use features such as auto-scaling).

I won’t go into the details of the VPC and Security Templates here, suffice it to say that with the multiple template approach the outputs from the creation of one stack were used as the inputs to the next.

For the Instance Templates the requirement was that the instances would be running Centos 6.3 and that we would use Chef to deploy the required application components onto them. When I started looking in to how we would set the instances up do this I found that the examples available for Centos and CloudFormation were extremely limited compared to Ubuntu or Windows. As this is the case I would recommend working from a combination of the Opscode guide to installing Chef on Centos and AWS’s documentation on Integrating AWS with Opscode Chef.

Along the way to producing the finished script there were a number of lessons which I will share with you to help with your installation, the first of these was the need to use a Centos.org AMI from the AWS Marketplace. After identifying the required AMI I tried running up a test template to see what happens before signing up for it in the Marketplace, in CloudFormation this failed with an error of ‘AccessDenied. User doesn’t have permission to call ec2::RunInstances’ which was slightly misleading. Once I’d signed our account up for the AMI then this was cured.

The next problem I encountered was really one of my own making / understanding. When looking at AMIs to use I made sure that we had picked one that was Cloud-Init enabled, in my simplistic view I thought that this meant that commands such as cfn-init that are used within CloudFormation to carry out CloudFormation specific tasks would already be present. This wasn’t the case as the cfn- commands are part of a separate bootstrap installer that needs to be included in the UserData Section of the template (see below):

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
 "#!/bin/bash -v\n",
 "function error_exit\n",
 "{\n",
 " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "ResFuseClientWaitHandle" }, "'\n",
 " exit 1\n",
 "}\n",<br /> "# Install the CloudFormation tools and call init\n",
 "# Note do not remove this bit\n",<br /> "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
 "cfn-init --region ", { "Ref" : "AWS::Region" },
 " -s ", { "Ref" : "AWS::StackName" }, " -r ResInstanceFuse ",
 " --access-key ", { "Ref" : "ResAccessKey" },
 " --secret-key ", { "Fn::GetAtt" : [ "ResAccessKey", "SecretAccessKey" ]},
 " -c set1",
 " || error_exit 'Failed to run cfn-init'\n",
 "# End of CloudFormation Install and init\n", 
 "# Make the Chef log folder\n",
 "mkdir /etc/chef/logs\n",
 "# Try starting the Chef client\n",
 "chef-client -j /etc/chef/roles.json --logfile /etc/chef/logs/chef.log &gt; /tmp/initialize_chef_client.log 2&gt;&amp;1 || error_exit 'Failed to initialise chef client' \n",
 "# Signal success\n",
 "cfn-signal -e $? -r 'Fuse Server configuration' '", { "Ref" : "ResFuseClientWaitHandle" }, "'\n"
]]}}

As the cfn-signal which comes as part of the bootstrap installer is used for messaging to any wait handlers defined in the template this can lead to long breaks at the coffee machine before any feedback is received if they are not present.

The final lesson was how to deploy the Chef Client and configuration to the instances. Chef is a rubygems package, so needs this and supporting packages present on the instance before it can be installed. Within CloudFormation packages can be installed via the use of the packages configuration sections of AWS::CloudFormation::Init which for Linux supports rpm, yum and rubygems installers. Unfortunately for the AMI we chose to use the available repositories didn’t contain all packages necessary for our build, to get around this I had to rpm on the rbel repository definitions before using a combination of yum and rubygems to install Chef:

"packages" : {
 "rpm" : {
 "rbel" : "http://rbel.frameos.org/rbel6"
 },
 "yum" : {
 "ruby" : [],
 "ruby-devel" : [],
 "ruby-ri" : [],
 "ruby-rdoc" : [],
 "gcc" : [],
 "gcc-c++" : [],
 "automake" : [],
 "autoconf" : [],
 "make" : [],
 "curl" : [],
 "dmidecode" : [],
 "rubygems" : []
 },
 "rubygems" : {
 "chef" : [] 
 }
}

Once Chef was installed the next job was to create the Chef configuration files and validation key on the instance. This was carried out using the “files” options within AWS::CloudFormation::Init:

"files" : {
 "/etc/chef/client.rb" : 
 "content" : { "Fn::Join" : ["", [
 "log_level :info", "\n", "log_location STDOUT", "\n",
 "chef_server_url '", { "Ref" : "ParChefServerUrl" }, "'", "\n",
 "validation_key \"/etc/chef/chef-validator.pem\n",
 "validation_client_name '", { "Ref" : "ParChefValidatorName" }, "'", "\n"
 ]]}, 
 "mode" : "000644",
 "owner" : "root",
 "group" : "root"
 },
 "/etc/chef/roles.json" : {
 "content" : { 
 "run_list" : [ "role[esb]" ]
 },
 "mode" : "000644",
 "owner" : "root",
 "group" : "root"
 },
 "/etc/chef/chef-validator.pem" : {
 "source" : { "Fn::Join" : ["", [{ "Ref" : "ParChefKeyBucket" }, { "Ref" : "ParChefValidatorName" }, ".pem"]]},
 "mode" : "000644",
 "owner" : "root",
 "group" : "root",
 "authentication" : "S3Access"
 }
}

The hardest part of this was the validation key, as we had multiple instances wanting to use the same key we decided to place this within an S3 bucket and pull the key down. During the script creation I tried multiple ways of doing this, such as using S3Cmd (which needed another repository and set of configuration to run) but found that using the files section worked best.

Once Chef was installed the client was started via the UserData section (basically a shell script), this then handed control of what additional software and configuration is installed on the instance to the Chef Master. How much Chef does at this stage is a bit of a balancing act as the wait handler within the template will fail the stack creation if its timeout period is exceeded.

As you can probably tell if you have got this far, the creation of the templates took quite a few iterations to get right as I learnt more about CloudFormation. When debugging what is going on it is worth remembering that you should always set the stack to not rollback on failure. This then allows you to access the instances created to find out where they got to within the install, as the UserData section is basically a shell script with some CloudFormation hooks, more times than not the faults are likely to be the same as you would see on a standard non-AWS Linux install. Also for a Centos install remember that the contents of /var/log are your friend as both cloud-init and cfn-init create log files here for debugging purposes.

After watching Werner Vogels keynote speech from AWS Re:Invent it’s clear that treating infrastructure as a programmable resource (i.e. using technologies such as CloudFormation and Chef) is somewhere organisations need to be moving towards and based on my experience so far I will be recommending using this approach on all future AWS environments we get involved with, even the small ones.

Whilst there is a bit of a learning curve the benefits of repeatable builds, known configuration and the ability to source control infrastructure far outweigh any shortcomings, such as granular template validation which I’m sure will come with time.

If you have any comments or want to know more please get in touch.

The subcategory called Big Data is emerging out of the shadows and into the mainstream.

Matt Wood with Robin Meehan

From left: Matt Wood, Chief Data Scientist at Amazon Web Services (AWS) with Robin Meehan, CTO at Smart421
Photo by Jim Templeton-Cross

What it is.

Definitions abound (who would have thought it? – quite usual in the technology market). For Big Data, we quite like the definition that originated with Doug Laney (@doug_laney), formerly META Group, now a Gartner analyst. It goes something like this:

 ” … increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources)”

Gartner continue to use this “3Vs” model for describing Big Data.

Unsurprisingly, others are claiming Gartner’s construct for Big Data (see Doug’s blog post, 14 Jan 2012).

Still confused?

Put another way, Big Data is commonly understood to be:

“… a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage,search, sharing, analysis,and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.” read more on Wikipedia.

Big Data could be executed on-premise if you have sufficient compute and storage in your corporate data centre. And some do, especially some large banks, and with good success. Several solutions are already out there on the market;  Oracle’s Big Data Appliance is just one example.  But it does also beg the question “why would you” ?

If you don’t want the CapEx of purchasing more tin, or don’t want to gobble up capacity in your own data centre, then there are alternatives. For example, a cost model now exists with cloud-based compute and cloud-based storage (for example, think of Amazon’s announcement of 25 percent reductions in the price of Amazon S3, it’s storage solution) that puts Big Data in the Cloud well within the reach of all UK enterprises. A cost model like that islikely to win friends in procurement and in corporate governance as well as in IT.

Hinging on technologies including Apache Hadoop clusters, Amazon Elastic Map Reduce (Amazon EMR) and others, Big Data is delivering a degree of analytics and visualisation not previously possible at affordable levels.

Don’t just take our word for it, ask around. We could point you to other experts in Big Data, such Matt Wood ( @mza ), Chief Data Scientist at AWS.

What it isn’t.

Big Data isn’t business intelligence (BI). What I mean is that Big Data isn’t BI in any traditional sense of the term. It is altogether another level on from that. Granted that some tooling enterprises may own may be recycled for use in Big Data analytics. But it isn’t another species, it’s another race.

Big Data isn’t a lame attempt at reviving a management information system (MIS); those should be left to rest in peace.

What it means for you.

By now, if you’ve read this far, something should be niggling away at you that you could be missing a trick. I trust it won’t be those voices in your head again. But it might be your instincts telling you how Big Data could answer those tough business questions – y’know, those “I can’t be asked” questions that existing systems just cannot deliver.

Now, you would not necessarily get our CTO to come right out and say that Big Data is the next big thing. But evidence we are assembling so far does seem to point to a new capability to deliver. For those with an appetite to understand their business in new ways, Big Data is delivering tangible intelligence that lets them see new dimensions, new possibilities and new revenue streams.

I did get a full radar lock on something our CTO said in the summer. It was a throw away line at the time but it stuck with me and with others. So, when the time came to consider an appropriate go-to-market message for our quarter three (Q3) focus, we decided to wheel out his one-liner as part of our messaging.

“It’s not about survival of the fittest -
it’s about survival of the best informed”
Robin Meehan, CTO, Smart421 Ltd.

Making no apologies to Charles Darwin or evolutionists, the statement is resonating with decision makers in the enterprise space, not least those in the Insurance sector. Why?  Well, we think it is because a lot of the big insurers operate under many names in their brand portfolios.

The capability to see and understand impacts of brand activities, such as Insurance Quotes, delivered using Big Data analytics in the AWS Cloud, is illuminating new gains that would otherwise have remained out of reach.

Don’t forget – brand analysis is only one use case for Big Data in the Cloud.

If the world is going Big Data crazy then you need to know what it is, what it isn’t and what it means to your enterprise.

Agree?  Disagree?

UPDATE 05 Dec 2012 – Our economist friend Tim Harford  (@TimHarford) sent this hilarious tweet: The data do not lie. OR DO THEY? Muah huah huah! http://dlvr.it/2b2NS1

UPDATE 06 Dec 2012 – Robin and colleague Ben Baumguertel (@bbaumguertel) are attending the Big Data Analytics event in London today (organised by @WhitehallMedia ).

Please Rate this post and Like this post below. If you can, please leave a Comment.

Along with Andy and Adam, I attended the annual IT Service Management Forum (itSMF) Conference in London on the 7th and 8th November 2011. This is regularly billed as the event of the year for service management professionals and everything ITIL.  It is supported not only by a range of delegates with serious interest in Service Management but also governing bodies, sponsors and 40 suppliers exhibiting their Service Management based tools, training and consultancy.  Of note, that may well be reflective of these times of austerity, is that the last time I attended the itSMF conference in 2009 there were over double the number of suppliers exhibiting at the conference with some notable leaders in the field not represented at this year’s event.

The main content of the conference were 50 presentations on a variety of subjects spread across the 2 days.  Typically the presentations ranged from thinly disguised vendor sales pitches, through theoretical rather than pragmatically based concepts, to real world examples of initiatives undertaken and the associated experiences. As a general rule of thumb it tends to be these real world experience based presentations along with targeted factual update sessions that provide the most value.

The sessions I went to varied greatly in their interest and usefulness with the main value coming from sessions covering:-

  • Developing Effective Performance Management, which although not a particularly revolutionary concept did help crystallise a few ideas regards a more system based approach for Performance Management.  Such an approach would be more holistic but provide differing levels of granularity in order that ‘default’ levels provided could be matched to respective audience requirements, whilst at the same time allow for drilling down to more granular based information as and when required.  One interesting idea was to utilise the Service Catalogue to identify the relative value of the Services offered to the business.
  • The ITIL 2011 Update session which outlined the major changes that have been introduced in the very recent publication of updated versions of the five ITIL manuals.  Whilst it is clear that we will need to reference all these new updated versions, approximately 80% of the changes are focused on the Service Strategy manual and cascading of these changes through the other manuals for consistency.  The only potential new concept is the introduction of a Continual Service Improvement Register, a concept we have advised clients on previously even if not using that specific terminology.
  • A session describing the currently under development ITIL Master Qualification. This will be the highest qualification attainable in ITIL and will be based upon validation of  the capability of candidates to apply the principles of ITIL in the real world. It will not be based upon any form of courses or examinations but instead will take the form of four stages working through Application, Proposal, Work Package creation and submission based upon real world utilisation and experiences before a Final Interview. There will be a set of 40-45 requirements that will be have to be met for candidates to be accepted onto the qualification programme, but further details are not yet available. Therefore the applicability to representatives from Smart421 is not yet known, but besides the personal value of such a high qualification there may be of value to Smart421 as a significant differentiator for Sales if anyone within Smart421 were to achieve this qualification.
  • Mobility, Big Data and Precognitive Support was a session presented by Chris Dancy,  the founder of ServiceSphere, which delivered an interesting and compelling view of the recent past, the present and what we should anticipate for the future. For example:-
    • Mobility:- There are 5.3 billion mobile subscribers (that’s 76 percent of the world population) with the growth being led by China and India.  Over 85 percent of new handsets are Web Enabled, which will facilitate continents like Africa to completely leap-frog  ‘hard’ infrastructure requirements like land lines and PC based networking. Mobile devices are already ubiquitous and will be the de facto integration method i.e. will increasingly take precedence and preference over ‘traditional’ PCs and laptops etc.  Facilitating integration will be primarily based around mobile apps,  one projection being that by 2016 cumulative mobile apps downloads will reach 44 billion.
    • Big Data:– a term the presenter thought will become as familiar as Cloud in the coming months and years.  Statistical examples included that the same amount of worldwide information generation that occurred between 1995 and 2000, was achieved in 2 days in 2010 and in 1 day in 2011. Another example was that the size of the internet in 2003 is the same size as Facebook alone  is today!
    • This amount of data is facilitating a move towards more precognition analysis etc. For example:- studying moods on twitter via automated textual analysis, happiness on foursquare mapped across cities etc. In addition to the presentation but another directly connected example, I heard a radio interview on the way home from the conference about perception of value for money of 3D films based upon automated textual analysis of blogs on film based sites. Back in the presentation these techniques are already apparently being utilised in extended news cycles by media organisations, with pre-event analysis occurring specifically in preparation for actual events – sounds a bit like pro-active trending!
    • Finally the speed of change that should be anticipated is already outlined to a large degree with some of the statistics highlighted above, but it is also argued that in the wider scale that “we are still in the first minutes of the first day of the internet revolution” and that Kurzweil’s Law of Accelerating Returns with the implied exponential and non-linear growth in technological capabilities and human knowledge is already occurring in line with Kurzweil’s 6 Epochs  ideas. Whilst it could be argued that such considerations are overly cerebral and abstract, it doesn’t take much consideration of the evidence and experience of the last 10 or 20 years to realise that the future opportunities and challenges are going to be a world apart from what we have seen previously.

Finally Andy & I also attended a couple of sessions based on management of ‘The Cloud’. We found both sessions basic and were left with the clear impression that Smart421’s capabilities and position are far more advanced than anything represented at the conference – so clearly this is an area we should be shouting about more!!

Recently, I attended the 3-day conference in London that combined, for the first time, EA and BPM, which had in previous years been the subject of separate conferences, see the overview at http://www.irmuk.co.uk/eac2011/overview.cfm for more details. Thanks to Robin Meehan presenting a session with Visa Europe we got a good deal on the ticket to go to all three days including the Seminar on Wednesday.

The first day gave me the opportunity to see the legendary John Zachman present a half-day introduction to his famous “Zachman EA Framework”. The seminar was subtitled “Enterprise Physics”, which made me think of Star Trek and Scottie the Engineer but maybe that’s just me. Zachman prefers using the terms “ontology” or “classification” rather than “framework” for the core 6×6 matrix (sorry, “normalised schema”) that compares with the periodic table that underlies the whole of chemistry. The main thrust of Zachman’s very entertaining presentation was that nobody can carry out any seriously complex activity without architecture and that architecture is the same for enterprises as it is for aeroplanes or one hundred storey buildings (but harder).

The analogies and application of EA to science and engineering showed how relatively young and immature is the whole practice of EA and Zachman can rightly claim to be a pioneer in the late 1960s and still going strong now at the age of 76. Robin Meehan wrote about him two years ago and I would echo a lot of the sentiments he expressed regarding the energy and passion he still displays.

In the afternoon on day one I attended a seminar on Business Process and BPMN, which told me that BPMN 2.0 has only four basic building blocks that result in 100 or so detailed objects with embellishments and decorations. For example there are something like 63 different categories of “event”. What BPMN 2.0 does is categorise into “common core” of just a few important fundamental concepts that can code the majority of simple business processes. There were a range of tool vendors in the exhibition supporting BPMN in various ways, many now based on standard archimate-style notation.

What surprises me a little bit is the way the business process delegates still seem to think they exist alongside EA whereas by definition EA encompasses the whole enterprise, as Zachman says “The whole thing including the business architecture and processes”, so therefore BPM falls within EA.

Day Two

Day 2 started with a nice opening by Sally Bean (@cybersal on twitter – Twitter was in evidence including tags #IRMEAC and #IRMBPM that I used for a bit) and Roger Burlton (@RogerBurlton) that focused on having a disciplined, coherent and shared architecture strategy that encompasses both EA and BPM; ok, I would argue EA already encompasses BPM but it’s good the similarities and overlaps are now being recognised and acted upon. The other statement that stuck with me was that “The common repository” is critical, something that causes a debate in our group with respect to federated SOA and autonomy of business units within an enterprise.

The keynote was given by Thomas Lawton (@TCLawton) who was clearly suffering with mild laryngitis so has to be applauded for getting through his description of breakout strategy, leadership and vision wheels so well. Some nice categorisations of businesses in frame of their response to the recession (Panic, Protect, Cloak, Conquer) and then in terms of breakout, being offensive (in the “attacking” sense in British parlance), i.e. “…the best form of defense is attack”. He spent a long time exploring the nature of growth opportunities, where Google are a “True Original” taking an emergent market by storm and Tesco are a “Big Improver” moving from laggard to leader in established market. I stopped to think about it and would probably categorise Smart421 as “Wave Rider”, not really a true original but taking on and leading the way in an emerging market (EA Consultancy). The only thing that bothered me slightly was the example in this space was Ryan Air – I’d like to think we have a much friendlier customer focus! Thomas’s “Vision Wheel” was an interesting concept, separating external and internal aspects and the final section was about how to create a “Magnet company” that excites markets and attracts customers. The key seems simply to build the Vision for the future based on the six aspects: Price, Features, Quality, Support, Availability, Reputation. I had a go at doing this for Smart421 below. It would be interesting to get other peoples’ views on the ratings.

Image

The afternoon keynote from Ian Gotts of Nimbus focused on CEOs and specifically selling BPM projects to CEOs. The first rule he quoted was “not BPM”, which was a theme of some other talks “Don’t mention architecture”. It reminded me of the famous football autobiography by Len Shackleton where he entitled a chapter “What the average club chairman knows about football” and left the page completely blank. Gotts’s talk used examples from the transformation of Carphone Warehouse from a basic “phone shifter” to a rounded customer-oriented gadget shop with supporting processes. The slides contained some interesting predictions like the market for BPM services to top $24bn in the next few years and he had a nice graphic showing an exponential increase in spending by 8 of their customers recently (could just be coincidence as business always increases over time). It was entertaining and made me more aware of how to present to senior business-people, as if I didn’t already know not to mention IT terminology.

Also today, I had the pleasure of attending two presentations by working Enterprise Architects from Shell and British Gas. It is always enjoyable hearing about real-world experiences that highlight gaps in the models. Dan Jeavons from Shell is far too youthful to know as much as he does about Enterprise Architecture but I found myself agreeing with what he was saying and it confirms my belief that implementing EA needs sponsorship from the top and there is a right way to do it (meta-model definition before tooling for example).

Jane Chang from British Gas pretty much developed her own practice, on the back of delivering Smart Metering to the company’s 10 million customers. The programme has been a great success and now has a large 400-person development team working on it to meet the architecture vision. A very good end to the day.

Day Three

And so to the third and final day of EAC and BPM and the obvious highlight was the presentation bySmart421 CTO, Robin Meehan and Chris Forlano Lead Enterprise Architect at Visa Europe on “Maturing Visa’s Enterprise Architecture Practice”.

Robin Meehan CTO at Smart421 (pictured left) with Chris Forlano, Lead Enterprise Architect at Visa Europe. Photo by Andrew Smale.

The session was appreciated by all and they asked some very interesting questions, like “How did you justify a 530 days budget for this work?”, which should probably have been answered by Mark had he been there.

Prior to that I went along to a Lean Six Sigma presentation and learnt a few more strings to use around promoting Quality through reducing variance (Six Sigma) at the same time as addressing the 7 Sins of Waste (Lean). I thought Peter Matthijssen was really good at using examples to introduce LSS as a practice for aspiring Business Process Architects and explained the concepts really well.

The morning keynote was probably the best talk of the whole conference by Jason Bloomberg on …. you’ve guessed it… The Cloud!   Or more specifically, “Architecting the Cloud – How EAs should think about Cloud Computing”. Both the Pros and the Cons were presented and the not so subtle message to delegates was to not let vendors drive down the route of private cloud and that public cloud cannot be trusted. I did think some examples: a Cloud employee taking a memory stick to your server and stealing your data, or the police impounding your (shared) boxes because of illegal activity by someone else was a little bit OTT. The main message reinforced our view that you must architect for the cloud and synergies with SOA were well presented, in particular the suggestion to extend SOA Governance to cover Cloud Governance, a reasonable extension as I’ve always thought SOA Governance should govern the underlying platforms for capacity and autonomy anyway. I didn’t quite get his point of Cloud services using REST couldn’t be governed as part of SOA because surely SOA is technology agnostic? His last slide on availability and redundancy with reference to the April Amazon outage provided a good discussion point afterwards and if anything this will be good for service providers like Smart421 offering experienced Cloud consultancy.

My second session of the day was “The Success of a Pragmatic Enterprise Architecture approach ‘STREAM'” by Jaap Schekkerman, Thought Leader Business Technology Strategy. I wasn’t completely convinced that these methods will work for everyone and the recommendation to design business methods on A0 format was provocative to someone like me who believes in a more componentised approach and that a process should fit on a single page to be understandable. Some of his slides also suffered from the A0 format and were incomprehensible. However, I did like Jaap as a presenter and he does have some original methods built into STREAM, which stands for:
Speedy Traceable Result-driven Enterprise Architecture (or Asset/Agile) Management, and it can be integrated with other frameworks and methodologies.

If I have one regret from this conference it is some of the session choices I made – Oliver Robinson’s presentation about improving the National Policing Agency drew a lot of praise, as did Tom Graves from Tetradian on “Respect as an Architectural Issue: a Case Study in Business Survival” but you can’t be everywhere. At least I have all the slides and further references like to tetradianbooks.com for the last one.

I admit I also suffered a little bit of BPM-fatigue after a while of going round the numerous vendors and trying to understand their products. However, if anyone has a need to deliver a BPM tool then I’ve got some good contacts now and a backpack full of literature and demos so give me a call or tweet me @smaley

Richard Holland, Operations and Delivery Director of Eagle Genomics

Richard Holland, Operations and Delivery Director of Eagle Genomics addresses delegates at the 2011 AWS Tech Summit, London 17 March.

The AWS Technical Summit in London 17 March [see http://aws.amazon.com/aws-tech-summit-london-2011/] was very worthwhile and no one could fail to notice that AWS themselves were taken by surprise in the exceptional attendance levels. Along with colleagues, Smart421 attended as AWS Solution Providers.

One of the key takeaways for me was how AWS continues to be highly responsive to the market. As well as bringing features to market in rapid succession, they have also listened and replied convincingly on the recurrent obstacle of security in the Cloud.

In fact, in addition to AWS’s already comprehensive security stance [see http://aws.amazon.com/security/] it emerged that one of AWS’s customers, Eagle Genomics based in Cambridge UK, had also permitted two independent IT consulting firms, AT&T and Cognizant, to perform ethical hacking on their AWS instances (permissions obviously required in advance).  The outcome?  Both firms reported that it couldn’t be done.

For hardcore sceptics (aren’t we all at heart), AWS say they have a list of partners that provide services around ethical hacking. This means that if your business case warrants a belt and braces approach on security, it’s possible to engage IT consulting firms these kinds of trials dedicated to your instance or instances in the Amazon cloud.

In reality, with some notable exceptions where systems are understandably internalised, both physical security and digital security offered in AWS is far in excess what the majority of organisations are able to provide for themselves. I’m certainly not alone in thinking that a far bigger risk for enterprises resides in the portability of data (e.g. copies held on company laptops, CDs, USB keys, etc) than in hacking instances of Cloud computing, particularly those on AWS.

It would seem that the objection around security in the cloud is being steadily eroded away. About time.

Next Page »

Follow

Get every new post delivered to your Inbox.

Join 1,122 other followers