The subcategory called Big Data is emerging out of the shadows and into the mainstream.

Matt Wood with Robin Meehan

From left: Matt Wood, Chief Data Scientist at Amazon Web Services (AWS) with Robin Meehan, CTO at Smart421
Photo by Jim Templeton-Cross

What it is.

Definitions abound (who would have thought it? – quite usual in the technology market). For Big Data, we quite like the definition that originated with Doug Laney (@doug_laney), formerly META Group, now a Gartner analyst. It goes something like this:

 ” … increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources)”

Gartner continue to use this “3Vs” model for describing Big Data.

Unsurprisingly, others are claiming Gartner’s construct for Big Data (see Doug’s blog post, 14 Jan 2012).

Still confused?

Put another way, Big Data is commonly understood to be:

“… a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage,search, sharing, analysis,and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.” read more on Wikipedia.

Big Data could be executed on-premise if you have sufficient compute and storage in your corporate data centre. And some do, especially some large banks, and with good success. Several solutions are already out there on the market;  Oracle’s Big Data Appliance is just one example.  But it does also beg the question “why would you” ?

If you don’t want the CapEx of purchasing more tin, or don’t want to gobble up capacity in your own data centre, then there are alternatives. For example, a cost model now exists with cloud-based compute and cloud-based storage (for example, think of Amazon’s announcement of 25 percent reductions in the price of Amazon S3, it’s storage solution) that puts Big Data in the Cloud well within the reach of all UK enterprises. A cost model like that islikely to win friends in procurement and in corporate governance as well as in IT.

Hinging on technologies including Apache Hadoop clusters, Amazon Elastic Map Reduce (Amazon EMR) and others, Big Data is delivering a degree of analytics and visualisation not previously possible at affordable levels.

Don’t just take our word for it, ask around. We could point you to other experts in Big Data, such Matt Wood ( @mza ), Chief Data Scientist at AWS.

What it isn’t.

Big Data isn’t business intelligence (BI). What I mean is that Big Data isn’t BI in any traditional sense of the term. It is altogether another level on from that. Granted that some tooling enterprises may own may be recycled for use in Big Data analytics. But it isn’t another species, it’s another race.

Big Data isn’t a lame attempt at reviving a management information system (MIS); those should be left to rest in peace.

What it means for you.

By now, if you’ve read this far, something should be niggling away at you that you could be missing a trick. I trust it won’t be those voices in your head again. But it might be your instincts telling you how Big Data could answer those tough business questions – y’know, those “I can’t be asked” questions that existing systems just cannot deliver.

Now, you would not necessarily get our CTO to come right out and say that Big Data is the next big thing. But evidence we are assembling so far does seem to point to a new capability to deliver. For those with an appetite to understand their business in new ways, Big Data is delivering tangible intelligence that lets them see new dimensions, new possibilities and new revenue streams.

I did get a full radar lock on something our CTO said in the summer. It was a throw away line at the time but it stuck with me and with others. So, when the time came to consider an appropriate go-to-market message for our quarter three (Q3) focus, we decided to wheel out his one-liner as part of our messaging.

“It’s not about survival of the fittest -
it’s about survival of the best informed”
Robin Meehan, CTO, Smart421 Ltd.

Making no apologies to Charles Darwin or evolutionists, the statement is resonating with decision makers in the enterprise space, not least those in the Insurance sector. Why?  Well, we think it is because a lot of the big insurers operate under many names in their brand portfolios.

The capability to see and understand impacts of brand activities, such as Insurance Quotes, delivered using Big Data analytics in the AWS Cloud, is illuminating new gains that would otherwise have remained out of reach.

Don’t forget – brand analysis is only one use case for Big Data in the Cloud.

If the world is going Big Data crazy then you need to know what it is, what it isn’t and what it means to your enterprise.

Agree?  Disagree?

UPDATE 05 Dec 2012 – Our economist friend Tim Harford  (@TimHarford) sent this hilarious tweet: The data do not lie. OR DO THEY? Muah huah huah!

UPDATE 06 Dec 2012 – Robin and colleague Ben Baumguertel (@bbaumguertel) are attending the Big Data Analytics event in London today (organised by @WhitehallMedia ).

Please Rate this post and Like this post below. If you can, please leave a Comment.

Caution workforce in the road!

What would your reaction be if the workforce in the road, fixing the road, did not have any tools or machines to do the job?

Frustration at the waste of time in the resulting traffic queue?

What would be your reaction if the washing machine repair man turned up without his tool kit, without a diagram of the appliance and without access to spare parts?

Refuse to pay the bill?

A security company providing security without enough staff

Questions in Parliament?

How is that so many Enterprise Architects can do their job without the tools of their trade?

Often Enterprise Architects are missing vital parts of their tool kit:

  • Standards
  • Principles
  • Reference architectures
  • Models of the Organisation
  • Application Landscape
  • Analysis and design tools
  • Information sources to feed the analysis tools
  • Stakeholder analysis

Worse than this they seem to lack the basic tools to be able to create the EA tools they need such as the processes to maintain the models, principles, guidance and governance.

Do you wonder why EA gets a bad name?

I am not suggesting that we go back to the old EA approaches

  • Boil the ocean documenting the current state
  • Tons of detailed standards (always out of date)
  • Heavy handed governance that increases costs,  misses deadlines and the point

And any of the other EA anti-patterns

Togaf 9.x of course points us at lots of artefacts and things to do, it is supposed to. We do not have to do them all, we can mix and match – What happens when we mix and match ourselves out of TOGAF9.x in all but name? Are we no longer doing architecture?

There are precedents for this situation:

SSADM was created and adopted, but everyone picked the bits they liked or could do. No one could afford to complete the whole SSADM – Especially with paper and pencil (there were few tools around).  SSADM became discredited; Every claim of compliance was subject to interpretation.

A similar thing happened to PRINCE.

I guess that there are many other examples of the dilution of the good practices until they are no longer effective.

Will this be the fate of TOGAF?

Are we architects no longer doing architecture?

…is governance of it.

Why is that? Well…

  • It’s where the organisational tensions come out – where the “rubber hits the road” of time scale and functional/non-functional compromises, i.e. normal ‘change’ life-cycle tensions
  • The main value is long term not short term (there are tactical opportunities though)
  • The value is opaque also, and difficult to define. We know it’s the right thing to do, but it’s tricky to keep that in mind when your latest project “pants are on fire”. Partially it is the opportunity cost of not making mistakes and so it intangible (i.e. when it is working well it might feel like you don’t need it and it adds little value)
  • It’s a cost centre, so easily attacked when there is a budget challenge, i.e. you need new skills/teams to deliver and manage that ESB, perform governance, service definitions, canonical model, transforms etc
  • SOA tends to be attached to one or two evangelists in an organisation, so when they move on or change role it can wither on the vine
  • It’s dull – not sexy. Organisations lose interest in governance.

It’s all about sustainability really…and the vibrancy of the governance process is a good leading indicator of the health of SOA in your organisation…so the success or otherwise of SOA in an organisation is intimately linked with the health of the processes driving business strategy, EA, etc.


Now I’ve had time to reflect on the Gartner Enterprise Architecture Summit, London 17-18 May (twitter hashtag #gartnerea), let’s recap starting with the closing keynote.

 The “Berlin Wall” between IT and the business – no, scrub that, –  IT and “demand-side stakeholders” (Kyte, session P5 closing keynote) still looms very large, causing IT practitioners to be apologetic rather than evangelistic about their craft. If the wall can come down, then a “reunification” can take place, but that will depend on CIOs and others being able to articulate IT in business parlance. And that will necessitate IT learning to talk about VALUE.

Only stakeholders can determine what they value, claimed Burke (session G7).  In other words, the focus needs to be more on the value received.

But is this thing called ‘value’ really so intangible and unquantifiable?

According to Kyte in his animated and charismatic closing keynote, enterprise architects must start with analysing full life Cost (estimating real cost of hairy ‘n hungry ‘dog’ not just cost of cute ‘n cuddly ‘puppy’), next enterprise architects need to analyse Value (taking ‘utilisation’ as a proxy for value), lastly they need to indicate Risk (expressing this in simple red / amber / green indicators). Present this as a 3-column report to a chief finance officer or anyone in the C-suite (Gartner-speak for board of directors) and they’ll get it immediately – unlike all other reports typical of the IT department.

Visualisations and models (Allega, session G13) such as Gartner’s new “Root Anchor” model, Business Capability model, Generic Federated model, TIME model, etc are all subservient to a Value Proposition, to answer “how will this be used to deliver a business result?” Perhaps the best way to communicate that is to use Gartner’s Business Value model, authored by Michael Smith (Lapkin, session G16), especially now that the focus for chief executive has changed in the last 12 months from ‘sheer survival’ to ‘return to growth’ (Lapkin, session G16).

 I listened carefully to others who I met complaining that the ‘framework bashing’ during many sessions delivered by Gartner analysts was over-stated and unnecessarily self-serving. Thankfully, Burke seemed to strike the right balance (Burke, session G7) advising delegates to be pragmatic, starting with Future-State Architecture (FSA) before ever touching Current-State and demoting any priority currently attached to selecting a framework or rigidly following a framework. “We’re ‘doing’ TOGAF” was not something analysts wanted to hear.

To address Future-State, and because organisations are increasingly “hyperconnected”, the trick for enterprise architects is to start thinking about “the lines between the boxes” (Robertson, session G23) and begin to work more closely with Sourcing and Risk, not just Security, as it may mean architecting-in some cloud-based capability.

 I debated some of the other conference themes in conversations afterwards with delegates. Among them, an enterprise architect called Carl Chilley, who also picked up on ‘Hybrid thinking’ hailed as a new discipline for transformation, innovation and strategy (Gall, session P2 opening keynote). This is derived from ‘Design Thinking’, which is now at the heart of Gartner’s EA thinking and modelling. Go to and search on the hashtag #designthinking. Chilley said that Gartner is in the process of changing a lot of their EA materials to reflect the implications of this world view. He said that there is a need towards “computing where people matter” (deliberate misquote of the E F Schumacher tag line to the book ‘Small is Beautiful: Economics Where People Matter’) where solutions needs to be people-centric.

Chilley makes a valid point. It certainly echoes the idea that value can only be determined by the demand-side stakeholders (i.e. the people in the business who use the systems that have been architected for them).

 If enterprise architecture is not about technology, it should be about “design [of] systems for humans”. Inspiring stuff for some enterprise architects who may want to take more of their cues from social science than from computer science. More Emile Durkheim, less Tim Berners-Lee?  You decide.

 It does make you think if there is more to ‘Design thinking’ that first meets the eye. During Gall’s opening keynote several books were put forward as suggested reading. These might interest enterprise architects:

 As far as the challenges posed by so-called ‘Wicked Problems’ those problems that just cannot be factored in and for which there are no apparent solutions. We pondered what an enterprise architect’s equivalent of an Icelandic volcanic ash cloud might look like…  a cloud of another type, maybe.

In a recent post, David Linthicum asks “Can SOA governance technology be distracting?“. His answer is yes, and he offers the following sound advice:

First, only purchase SOA governance technology, if it’s indeed needed, after you have a complete semantic-, service-, and process-level understanding of the problem domain. Never before.

Amen to that. In my opinion, for all but the most mature and involved environments, the procurement of an SOA governance platform should be well down the list of priorities. I’d add to David’s list of things that need to be ‘worked out’ before you get that cheque book out:

  • What is your vision for governance itself? Do you want to adopt a ‘iron fist’ or ‘hand in glove’ approach? Is your registry going to be a mechanism for governing or a side effect of it?
  • Who’s going to populate it? Have you got your analysis, design and development processes sufficiently honed that your repository isn’t going to turn into a dumping ground of candidate services?
  • Have you actually got any services live yet? Governance is a whole lifecycle thing. Until you’ve worked out how you’re going to deploy and manage services in the production environment and demonstrated that this works, how do you know what capabilities your governance platform needs to offer?
  • Most importantly: What are the use cases for your governance platform? Can you demonstrate that these use cases can’t be addressed using your existing tooling (even if that’s Microsoft Excel)? Be honest with yourself about when you’re likely to implement these use cases. If the answer is further than one year away, then for the time, you might be wise to forget them. There is little point in spending good money on runtime governance or automated deployment technology when in a year’s time you’ll be able to get more for less.

A lot of projects using SOA governance tools at the moment treat them as glorified databases. If that’s where you’re at, consider using something less specialised that allows you to evolve your ideas, understanding and schema before you commit to something that will make this innovation harder and more time consuming. When you’ve spent six to twelve months getting your ducks in a row, so to speak, you’ll be in a much better place to make decisions.

I’d really welcome stories from people about how they’ve implemented governance platforms in the past, whether they’re informal (e.g. Wikis, bugtrackers, spreadsheets) or formal (e.g. IBM Websphere Registry and Repository, CentraSite from Software AG): What did you implement? What worked? What didn’t? What would you do differently next time?

People have often asked me “what is the value of (enterprise) architecture?”.  Well, first off, let’s think what architecture is, so we can talk about the value it provides.

Architecture is really a best practice, following the principles of good design from a top-down approach, whereby some sort of concept or vision is sketched out, which is firmed up into a blueprint, before the design is formulated using the applicable standards and policies needed to meet requirements, and finally, the artefact is built.  It doesn’t really matter if you are designing a house, an application or the whole enterprise – the same basic process is followed.

By following this approach the beginnings of a best practice start to emerge, which should deliver the following benefits:

  • Standards and statutory requirements are identified
  • Redundancy, duplicity and waste is minimised
  • Resources can be reused and operations and maintenance are streamlined
  • Return on investment is increased and total cost of ownership is decreased
  • A plan for future development emerges, which can be consistently communicated, and to which all interested parties can participate

That’s all good and said, but how do you ensure these benefits are delivered?  Simple – that’s the role of Governance.

What governance does is makes sure that your architecture is effective and sustainable.  When an architecture is initially defined, many architects make the mistake in that their designs are efficient, but this is not the same as being effective.  Therefore, the governance model must ensure that what the architecture delivers is what the business and IT communities really want, and is being utilised to its maximum potential.

Architecture also suffers the problem in that after a period of being in place it is not always as it was when it was initially designed, because changes to the realised architecture have not been reflect back into the design.  Or on the other hand, the requirements or technologies have changed over a period of time, but the designs have not, thus rendering the architecture ineffective.  This is what we mean by sustainability – the architecture must be effective over a measurable period.

And this is the value of governance to architecture – it ensures effectiveness and sustainability, through its quality review processes, so that compliance and uptake can be measured, and where necessary, the architecture amended.

But the governance model itself must also be effective.  It’s no good putting in place over-bearing governance processes that make it too difficult for people to comply with your architecture – all this will do is stifle creativity and development.  Or worse – give people an excuse for not adhering to your architecture.  What you should instead be doing is encouraging people to increase the uptake of your architecture i.e. make them see the benefits of governance, not the restrictions.

Therefore, your governance model must be complementary to your architecture; that is, as the maturity of your enterprise and it architecture increases, so too must the maturity of governance.  This is one of the objectives of governance that many people overlook – your governance program should actually be increasing the maturity of your organisation and its approach to architecture.


Get every new post delivered to your Inbox.

Join 1,122 other followers