SOA isn’t dead, it’s barely out of nappies

On the oft-discussed “SOA is dead” theme – surprise surprise I don’t agree. I think maybe that “SOA hype is dead” is a fair perspective and it is now entering the “business as usual” phase with many but definitely not all of our customers – every early or middling-adopting customer now knows it’s a good idea (the best idea out there at the moment) for the right use cases and is struggling to varying degrees to make it happen and get the benefits.

I’ve revised my view of the payback period – if I can generalise, I now think SOA is a 15 year endeavour, with services gradually building up over time and so reuse getting better over time, and organisations learning from their mistakes as they go also. The payback period is shortened by doing the right things – understanding your business architecture, and hence investing in the right services, making them reusable, good service development and life-cycle hygiene etc.

So the reasons for doing it are just as valid as ever – we’re just now in the hard slog phase and so the marketing flies have moved on to fresher…er…pastures.

 

Open Source Middleware (a.k.a. ESB) Choices – Part 1

In part 1 I set out my requirements for evaluating and choosing an open source ESB, and promptly fall down a rabbit hole of OSGI and Maven when I get to see how much more there is to them than I was previously aware.

From time to time we get requests on how to get started with middle-ware technology on the cheap. Here the emphasis is just about connecting service providers and consumers up, without getting into anything fancy like orchestration or service repositories. Of course, in an ideal world the solution should not rule the latter out.
So here are the requirements I can filter out of a few of these conversations.

• Open source for low cost and wider options when upgrades appear – i.e. not always forced onto the latest version.
• Must handle WS-Security, WS-Addressing
• Freedom for choosing java-XML binding framework
• Supports contract-first services design (as opposed to generating the service artefacts (WSDLs, schemata) from java classes.
• Run-time is ‘light’: i.e. when service-enabling components are deployed on the same machine as an application which will be service enabled, these service-enabling components do not gobble up all of the resources.

Contract first development is very important in a heterogenous environment. See arguments on the object – XML impedance mismatch here. Another way of putting it is: if you are going to stick with just one language (e.g. java) then why bother with XML in the first place – just go with some RMI technology, RMI-IIOP. If we are using Web-services, then interoperability is a big consideration, and for that we have to think contract-first.
One of the reasons for separating out the java-XML binding from the web-service end-point binding code is that it is great to use the same pojo to describe an entity, whether it is serialised to XML or persisted to a database or just as a value object.
On the one hand it is good practice to work with web-services in a contract-first style, and on the other hand: if you use the code (specifically the domain objects to which the XML is bound) throughout your application then you can introduce a dependancy on the XML marshalling generated classes, which is not great either. In an automated build environment, it means building your gen-src directory from schema before you can write any src code which uses it.
In the past I have got around this by specifically generating class libraries of domain objects, using jaxb, and then importing the resulting jar into any code (both client and server side) which manipulated these objects. The compromise at the time was that I ended up writing my own endpoints (servlets) to expose web-services – which is OK when there is not much WS-*, (e.g. Addressing, Security) going on.

I wanted to see if the latest incarnation of the open source frameworks would enable contract first, generation of domain objects first (such that they could also be used in persistence layers and as value objects) and relatively easy handling of WS-Security and WS-Addressing.
The new kids on the block for me are ServiceMix (a.k.a. Fuse), apache cxf, spring-ws, and the sun JAX-WS.

The previous time, the players had been Apache Axis, WSIF, and JAX-RPC. Oh I almost forgot Castor. Every one of these had had their own java-XML binding code and none of the produced beans were interoperable between frameworks. Stand-alone java-XML binding frameworks like JAXB (1.x) were not interoperable with the web services binding (e.g. JAX-RPC) generated objects.

Anyway: enough of the background… The first two I wanted to look at were both FUSE and Spring-WS, as they both allow contract first (Spring-WS will not allow anything else) and they both support Spring and its IoC (another Good Thing, but that’s a different discussion).

I had only got around to looking at FUSE when I fell down the first rabbit-hole: OSGI and Maven. I have had a look at the excellent video tutorials by Adrian Trenaman of progress (formerly Iona) software, (see the demo videos tab at the bottom of this page.
I had been aware of Maven for a while as a ’slightly better Ant’, but the demo and a bit more digging around reveals there are two big extra features in Maven which move the game on a whole lot more:
Firstly there is the project template feature. This is the feature whereby you can create a java project using a Maven build file with a command-line argument. The command builds the appropriate directory structure and even the correct POM.xml (Maven equivalent of an Ant build.xml file). Although I had been demo’d this before, it has only really sunk in this time what a big deal this is.

We have in the past put a lot of energy into our automated build system, based around Ant. For it to work well, there is a mandated project directory structure, and set of files and libraries have to be in the right places relative to each other. There’s a bit of a learning curve on top of Ant to understand what is going on. The template projects from Maven give you all that in one go. That becomes especially evident when you try a new project: for example an OSGI plug in project. You just run Maven with the proper archetype and bingo…

Secondly there is the repository system. You can configure a set of remote repositories, and just by specifying a library in your project file (.POM file) the maven build will try to fetch the library – of the version you specify too to your local machine, in a local repository – which is then shared amongst all of your projects. Again you notice how powerful this is when you download a maven-enabled project, and on the first build it goes and fetches all of its libraries which it depends on – unless you already have them locally. A large number of common shared libraries (e.g. most of the well-know apache projects) are available in the default repositories. It is possible to configure which external repositories are trusted and should be used.

The repository system has become effectively just another resource, to the extent that to install an OSGI bundle from the OSGI console (more on this next time) ‘mvn:’ is named as the protocol type for a given resource. The resource is then seamlessly retrieved; either from local storage, or from one of the configured remote repositories.

All clever stuff.

So from starting to look at Open Source middle-ware, I have fallen down a couple of rabbit holes. The maven excursion is definitely going to make me sit up and give that a much closer look (talk about late adopter!). The second rabbit hole for me was OSGI. more on that next time. Then it will be back on track for the open source middle-ware.

 

Organisational structures and challenges for integration teams

Q. How have organisations approached setting up an Integration Competency Centre (ICC)/Centre of Excellence (CoE) and what are the challenges?

A. An ICC/Integration CoE (put your organisation’s favourite name here) includes the following functions, which sometimes organisations have in separate teams, and sometimes combined…

• Standards creation and ownership
• Integration project governance, including the admin aspects
• Service definition and canonical data modelling, owning data representations ‘on the wire’ etc
• Doing real work for real projects, including the project/resource management aspects of this

Different organisations approach this in different ways. The key questions to consider include:

• Funding model – centralised or decentralised (i.e. project based)?
• Organisational model – if your organisation is heavily de-centralised (typically it’s grown through acquisition and the business strategy has been to leave acquired units to behave autonomously) then the ICC structure needs to reflect this in order to succeed.
• Change PMs – are they rewarded/incentivised for supporting reuse? for minimising operational (IT and business) as well as change costs? If this is not well set up then you’re in for a continual battle.
• Use of external product vendors – if they do not insist on the SOA standards/architectural guidelines role being fulfilled, this is an indicator that they don’t know what they are doing!
• Are you expecting to use the supplier’s data model for integration?
• Are you using significant offshore/remote resources? – this strongly affects how best to structure your teams

So what’s the best model? Well – of course – it depends. In general we recommend a decentralised team with some limited centralised standards support/expert help and project management/admin capability as this reflects the reality that we see with our customers – i.e. that a large non-project cost centre is not sustainable. The key is in the sustainability of the relationship between the ICC/ICoE and the individual project teams/federated parts of the organisation – if you get that functioning well with the projects feeding back ‘value’ into the ICC and the projects see the value that the ICC is providing (by removing the barriers to integration), then you’ve cracked it. This is all a subset of our ‘Sustainable Enterprise Architecture’ vision really.

Other challenges to consider

Here’s a hit list of other things to think about/look out for that we might have seen out in the field…

• Holy wars – over technology etc, partially due to poor comms/selling of pros/cons/vision typically
• Keeping up the energy/commitment in the long run – it’s a 10k run, not 100m sprint. It’s always difficult for any long-term initiative to survive business cycles and staffing changes.
• Business involvement/input – depends on whether it was a “technology sell” in the first place – but fundamentally the business guys should know best about where various services are likely to head, and therefore maximise agility down the road
• You can only go so fast, each step on the road takes a while to bed down, and you can’t leap-frog a maturity step, so allow for bedding-in phases. A common issue we see is that initial versions of internal standards are “over-egged” making it hard for service providers to comply with them – maybe just intellectually difficulty if not technically so. There maybe a bit of “showing off” going on there from new staff in the ICC who are keen to impress etc. Just stop it, right? I’m not interested in your use of the emerging esoteric WS-yada-yada standard when you don’t even have a service catalogue under control )
• How do you manage a tier 1 supplier? – SAP, Oracle etc – to ensure you buy the right products for you, even though their consultants are all over you like a rash. A common and tempting pattern for organisations is to go for a “one stop shop” from a single vendor for HR, ERP etc approach so that they can minimise integration problems. I’m not saying that this doesn’t have merit, but the key point is that your integration standards need to still be yours if you want to retain any control over the selection and integration of other components in the future. Ensure your suppliers have the necessary motivators/vested interests to deliver ‘good’ services that integrate well with the wider solution (i.e. they they don’t have a cost driver to minimise the customisation of their product leading to non-ideal/non-conformant services). Also if you have multiple partners/vendors involved, ensure that they have no incentive to rubbish and impede each other. As a customer, that’s no fun.
• Be clear about deliveries from your integration partner – exactly what they will produce and how it will add value, not just trust them to come up with the rigour/’magic of the ESB’, as there’s a lot of BS out there around integration/SOA.
• Service versioning – We tend to favour an “n+1 model”. but often we see that organisations don’t sort this out early enough as they don’t see it as a “day 1″ problem and then falter later on.
• Service ‘hygiene’ in general – invest in it.
• Metrics to measure and demonstrate success and business benefit, and also the effectiveness of governance processes (Robin’s rule – if your governance process never “stops” a project, it’s a sick process)
• Have a communication plan – your heavily federated organisation isn’t going to guess what your ICC is up to are they and magically fall into line with it? And probably the key point here – keep it alive. One time comms happens – well – once. Once you’ve read this blog you’ll forget it in 15 minutes, if I’m lucky. Your ‘virtual ICC’ colleagues will do the same unless you keep repeating the messages. It’s the old adage – tell them what you’re going to tell them, then tell…etc.
• Plan for the necessary admin/management support for your ICC – especially important for the ongoing governance process/project touch points as this task won’t happen for ‘free’.
• Define a concessions process – this is the get out of jail card – grant concessions to deviate from standards in special cases, document concession & mitigation/consequences, set a review or mitigating action deadline.
• Don’t let the ICC get bogged down in delivery, so not not enough time is spent on vision/standards/governance. A tough one to avoid this and where lots of organisations struggle. It all comes back to good project planning/resourcing really.

That’s all there is to it!

 

WebSphere Integration User Group highlights

After a journey characterised by a conspiracy between parking meters and failed tube signals, I made it yesterday to IBM’s Hursley Park for the WebSphere Integration User Group meeting. Here’s the photo at a sunny Winchester station as proof…ok…so Jamie and I forgot to take any photos at the event.

The key note presentation was from Kevin Turner – ESB and Messaging Technical Strategy Manager. He covered IBM’s Smarter Planet initiative and then moved on to future architectural trends – the noteworthy points were:

• The 2nd wave of SOA is coming – there’s a significant number of ‘1st wave’ adopters out there now who have been through the joy and the pain, and have now fully understood the upfront investment required and the potential benefits. This 2nd wave is likely to consist of organisations trying to join up their islands of SOA (probably built along departmental lines due to a sensible pragmatic strategy of ‘baby steps’ SOA), and so federation of multiple ESBs will be a key theme. Governance will be crucial here if these islands are ever to be consolidated into a virtual enterprise bus that solves the problems of visibility of services across ESBs, end-to-end security and manageability etc.
• Patterns – IBM are working on building some integration patterns support into their tooling (presumably WebSphere Business Modeller, WebSphere integration Developer etc) to allow an expert from an organisation’s ICC/ICoE to define the standard ‘pattern’ and therefore accelerate the development of instances of that integration pattern. The integration developer might just have to supply the pattern parameters for example, with many of the decisions such as how to manage errors etc already consistently solved for them.

There were a couple of presentations about specific MQ-related Supportpacs (MO71 and IH03) which I expected to be dull due to their nature, but the presenters managed to bring them alive – especially Paul Clarke. It was clear that he’d written and refined quite an impressive beast of an application over many years and was rightly proud of it.

Kevin mentioned CloudBurst during his keynote, and I managed to get some time with an IBMer later in a hastily arranged session to discuss it some more – I’ll post details in another blog post later as there’s quite a lot to report.

That means I missed the start of the next session about the developments in the WebSphere Process Server product to better support the human interaction aspects of BPEL processes (which have been sorely needed in our experience!). Paul Smith demoed the Business Space web app from WPS v6.2 which goes some of the way to addressing these shortcomings, with better human ‘override’ control of processes (skipping a step, repeating a step etc) and better visualisation tools for business users to use to understand where a particular process instance has got to, etc. This is clearly still a developing area of the product set though.

The last session I attended was a demo of rules from the recent iLog acquisition by IBM. An ex-iLogger Lenny Bromberg gave a very engaging demo which involved my colleague Jamie playing the role of “business user” to dynamically change rules that influenced the behaviour of a mock motor insurance quotation app. An interesting aspect of Lenny’s “pitch” was that essentially rules engines are 10 a penny, there’s several good open source ones out there if you want one, but what rules provides is a Business Rules Management System (BRMS), i.e. all the significant extras wrapped around the actual runtime rules execution environment that you need to make it really workable, manageable and governable. This includes rule development/editing environments (developer IDE and business facing), rule versioning, rule comparison tools, audit, simulation environments etc. Some other observations:

• Lenny’s experience from previous projects where they have integrated with BPM solutions (like WPS etc) is that they often find that the business process definition/BPEL has become ’spaghetti’ as the business rules are not clearly separated from the business process – and so the use of an external rules engine enforces a good business process automation design practice, and leads to more maintainable BPEL etc.
• This is related to BRMS’s in general and not specifically iLog rules, but a weakness that I could see is that the rules rely on a good, stable business object model and we know from experience with numerous customers how difficult it is to get enterprise data models together and agreed. This is the potentially shifting sand that the rules are all built upon.

Many thanks to Mike Wyvill and Keith Guttridge and others for organising the event. Well worth the £28… )

 

BizTalk Server 2009 (2006 R2) Utilities

BizTalk Server 2009 was released end of April 2009. This tends to lead to  a natural  review of our internal handbooks and the utilities that complement our processes.

When approaching all BizTalk engagements we  initially consider the SOA Roadmap and development methodologies and test frameworks that support these approaches. The majority of the utilities/extensions mentioned here are available as open source, or free trial download, but rarely do I see these utilities combined. It is assumed that these are applied to the traditional base build of Windows, SQL Server, optional WSS and BizTalk.

WCF LOB Adapter SDK SP2 (http://www.microsoft.com/downloads/details.aspx?FamilyID=47AB6F21-0D8B-4C90-A8B9-E8647281B164&displaylang=en )

The LOB adapter pack has now been extended to support SQL Server and is worth considering when looking at using WCF bindings.

BizTalk Adapter Pack 2.0 (http://www.microsoft.com/downloads/details.aspx?FamilyID=76736ba7-3c05-4436-9353-1c33f9005194&displaylang=en ).

This is a 120 day evaluation extending the WCF LOB Adapter SDK, to enable the auto generation of schemas and ports for the additional new bindings. There is an additional cost for this pack, depending on the licence model of your client.

Nunit 2.5  (http://www.nunit.org/index.php?p=download ).

Still popular with clients and used  to manage BizUnit test cases. The more traditional option is mstest, but useful to include for completeness.

BizUnit 2.3  (http://bizunit.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=9001#DownloadId=23581 )  or 3.0 (http://bizunit.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=20013)

BizUnit is an excellent declarative test framework for managing test cases and for generation of automated regression test packs. The newer release 3.0.x.x has additional feature support. Some clients like to make use of the excellent BizUnit Designer, which provides a UI for test case generation over editing raw XML. This is useful for early adopters, helping with understanding of what features are initially available in the framework.

Microsoft BizTalk LoadGen 2007 (http://www.microsoft.com/downloads/details.aspx?FamilyID=c8af583f-7044-48db-b7b9-969072df1689&DisplayLang=en)

An additional tool to coordinate the execution of performance and stress tests.

BizUnit Designer 1.4 (http://bud.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=12968#DownloadId=33409) used with BizUnit 2.3

 A useful tool providing a UI to assist with the initial creation of test cases.

BizTalk Deployment Framework  (http://biztalkdeployment.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=17826#DownloadId=54972 )

This is a must for any development team. Having used the older framework extensively, the new features support msbuild projects over nant. This is a real time saver when managing complex build and deploys.

HTMLHelp.exe ( http://www.microsoft.com/downloads/details.aspx?FamilyID=00535334-c8a6-452f-9aa0-d597d16580cc&displaylang=en )

A Pre-requisite for the HTML output of the following orchestration profiler/documentor

BizTalk Orchestration Profiler 1.1.1 ( http://biztalkorcprofiler.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=6375#DownloadId=17235 )

An excellent tool for verification of code coverage of orchestrations in BizTalk. Generation of a help file analysing orchestration performance. Used during test cycles for verification of regression test packs coverage.

BizTalk Documentor v3.2 (http://biztalkdocumenter.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=8689  )

A useful tool for documenting the configuration of a BizTalk implementation. Useful to include in the deployment and release cycles for configuration management issues and to share with support teams, to verify configurations.

BizTalk Server Best Practices Analyzer v1.2 ( http://www.microsoft.com/downloads/details.aspx?FamilyID=93d432fe-1370-4b6d-aaa8-a0c43c30f5ab&displaylang=en)

An essential tool for all client deployments to generate and understand compliance reports. The latest version supports BTS2006, 2006R2 and 2009.

MessageBoxViewer ( http://blogs.technet.com/jpierauc/pages/msgboxviewer.aspx)

An invaluable tool for querying and analysing system configurations, especially for warnings of potential performance issues. This is more of a dynamic analysis than best practices analyser.

 Each of the downloads provides excellent detail on how to use the individual utilities. These are the utilities that we use to support our SOA Development Methodology when implementing and supporting BizTalk Server implementations. They provide the basis for development, test and deployment frameworks.

 

A barrier to internal metering of service calls?

One trend that we’ve noticed is that organisations are typically very poor at organising themselves to create good cross-charging schemes for the supply of internal IT infrastructure or IT shared services, and often only have a very coarse-grained view of what the provision of these services really cost. At an individual change project level, this makes it impossible to make educated judgements about the likely ‘run’ cost of a solution, and so the architectural trade-offs that have to be made must be sub-optimal.

An explanation I’ve heard in the past for this is that the internal accountants just don’t “get it”, but that’s never rung quite true with me. I read an interesting suggestion on the cloud computing Google group for another reason – that if internal IT managers did accurately define the costs of internal shared IT service provision then they would be opening themselves up for direct comparison with external providers of equivalent services, so it’s a defensive mechanism basically. They don’t want a Cx0 coming to them saying “OK – you change me £x per service call, but I can get them for £y from [substitute your favourite cloud/SaaS etc offering here]“…

 

IBM WebSphere CloudBurst appliance launched

We first heard about this appliance at the UK WebSphere User Group meeting back in March although it was commercially sensitive at the time so I couldn’t blog about it, and we’ve been waiting for it to be launched. IBM are leveraging their acquisition of an appliance format with DataPower to bring other appliance products to market of which this is the first. It’s an interesting product, aimed at both those looking to create a private cloud and for the public cloud vendors.

It provides the ability to dynamically deploy and manage virtual machines including the new hypervisor version of WebSphere Application Server (which supports the OVF standard) including security and chargeback facilities.

I’m looking forward to our internal WebSphere practice finding out more and getting to grips/playing with it.

See http://www-01.ibm.com/software/webservers/cloudburst/ for more details. What’s the next functional area that is ripe for ‘appliancisation’ (if that’s a word…)?

 

What should you call your SOA project? Answer: something else

Whilst travelling across a few counties in the UK I found some time today to catch up on the miriad of email newsletters etc that I get, and picked up on an interesting point in a zapthink article. One point made was that rather than badging your project as an “SOA project”, that it’s better for it to have a business meaningful name. This got me to thinking about the various SOA-related projects that I’ve seen our customers execute, and how, if they get into trouble (fail to deliver on benefits, technology issues etc), then SOA typically takes the bullet.

So how about some other candidate project names such as:

• ‘Business area X’agility enablement project
• ‘Business area Y’ change cost reduction programme (there must be a snappy acronym hiding here – how about cost reduction advancement project )

What I like about this approach is that the business objective is then placed first and foremost in the minds of the participants – it sounds trivial but it is so easy to lose sight of the real objectives when you are worrying about your service versioning strategy and a pack of vendors are clamouring around you with their products…

 

UK WebSphere User Group March 2009 – Highlights

As promised, here’s some notes from yesterday’s UK WebSphere User Group meeting (combined with the UK WebSphere Integration User Group) – held at IBM’s offices at Bedfont Lakes.

First of all – let’s start with a picture of the happy crew at our stand. Rohima did an excellent job hunting down victims to force business cards upon, with me and David Taylor in support.

The keynote presentation was by Rob High, the IBM Chief SOA Architect, concerning the 2009 technical strategy and directions for the WebSphere portfolio. Unfortunately (for you – the reader), much of what he said came with a “not for the public domain” health warning. Interestingly he was still banging the SOA drum and when I asked him later in the Q&A session about it he was quite dismisive of all the recent ‘SOA is dead’ dicussions on the web – he gave the message that I wanted to hear which was essentially this – there’s nothing new under the sun, good integration practices are still good, and so the current hype cycle status of SOA should not stop us from still understanding the fundamental business services of an organisation and supporting them with technology in an agile way. I think there is some marketing difficultly with the term SOA now maybe but there was no hint of IBM moving away from it.

Rick Robinson presented on Web 2.0, going through quite a bit of background material and then mapping that on to IBM products, and their support for REST, OpenAjax and Dojo. Whenever I attend this kind of presentation I always pick up a few Web 2.0-ish sites/things that I hadn’t seen before – I guess this is the nature of the relatively viral nature of the subject itself. My favourite was http://www.twitscoop.com/ – which gives a real time view of Twitter topics that are being discussed as a tag cloud. Another one mentioned was http://brightkite.com/, a social networking site that was location-aware before Google Latitude came along.

I then attended another session from Rob High about EA (enterprise architecture) and the relationship with BPM (business process management). For me this was more of an EA revision session (an IBM view on TOGAF to some extent) and I didn’t get so much from it, except the IBM roadmap for their recently acquired TeleLogic System Architect product and how that fits into the roadmap for WebSphere Business Modeller. The vision is that they will remain separate tools but eventually with a shared repository. Interestingly none of the Rational modelling tooling was mentioned apart from saying that it was focused on software rather than business modelling, so there was no vision to merge System Architect into Rational Software Architect in anyway.

One other thing Rob mentioned was an expansion of that massively overused phrase “IT-business alignment” (doesn’t every IT initiative that comes along promise this?!?) into more several more defined levels of alignment – this rang a bell for me and it’s something I’ll look into a bit more I think…

Whilst my colleague was presenting a DataPower case study from a customer project in another room, the final presentation I went to was a full on techy session from David Currie of IBM about the new features in WebSphere Process Server (WPS) and WebSphere Enterprise Service Bus (WESB) v6.2. I wanted to keep on top of where these products are going at a detailed level. The degree of change in each release is pretty amazing, but also leaves a slightly sour taste in your mouth as you realise that IBM are plugging feature gaps in the product that sometimes you knew about (e.g. code assistance to help deal with the SDO model) and sometimes you didn’t (ability to act as a service gateway – which seems a pretty fundamental thing for an ESB to offer). Apart from the fairly extensive changes to support service gateways (a number of new mediation primitives etc), the developments in the human tasks aspect of it are the most interesting to me. You can now attach documents to business processes, and users can override a business process flow (sounds dangerous! – but intended for those processes where business exceptions/interruptions can occur at any time). Sorry to finish on a negative, but one obvious gap was that SOAP 1.2 is now supported, but not for SOAP over JMS – purely due to them running out of time to get it into the release AFAICT.

So – all in all, a good user group meeting, especially as it finished with beers. Many thanks to Rick Smith and co for organising it.

 

Agile and Enterprise Software Development

Having recently re-read Alistair Cockburn’s book on Agile Software Development, I thought I’d repeat some of the concepts that he describes and try to expand on how those may apply to the development of Enterprise-level software, compared to Application-level software.

There is a very interesting section describing concepts for the design of a methodology for software development. Within this, the choice of a suitable approach is based on a range of criteria, including the following:

• Methodology Size – the number of control elements in the approach
• Ceremony – a level of precision and tolerance applied by the approach; greater ceremony corresponds to tighter controls
• Methodology Weight – the product of size and ceremony, for comparison purposes
• Problem Size – the number of elements in the problem domain and their inherent cross-complexity
• Project Size – the number of people whose efforts need to be coordinated
• System Criticality – the importance of the system, or the level of damage that may by caused by any undetected faults; Alistair uses a scale as follows: loss of comfort, loss of discretionary money, loss of irreplaceable money, loss of life

Other criteria listed in his book are precision, accuracy, relevance, tolerance, visibility, scale and stability. This last one is to determine how likely things are to change, explained with a suitable statement (‘if I were to ask the same questions today and in two weeks, how likely would I be to get the same answers?’)

Those I have included in a bullet list above are perhaps most useful to classify the different types of methodologies, with the others being useful to expand on the detail of any particular approach.

The most commonly-used Agile methodologies, such as XP and Scrum, are inherently lightweight, having minimal control elements and low ceremony, with them being most suited to smaller problems and projects.

Alistair includes an interesting table showing how projects can be characterised by communication load, criticality and priorities. Discussions can then position which type of methodology should, or should not, be used for a project under consideration. Those projects with smaller teams, changing requirements, flexibility in delivery, low risk and small scale are seen at one end of the scale, whilst massive programmes and projects needing strict controls against high risk (financial or life implications) will appear at the upper end of the scale.

In many cases, development of a new piece of standalone application software can benefit from the advantages of an Agile methodology. In this, early visibility of functional software is really useful, allowing changing ideas of the purpose and features of the application to be changed as development progresses. That has to be the true home of Agile software development, backed-up by approaches such as Test-Driven Development to improve the quality of software from ‘day one’.

However, it would seem that for the development of multiple integrated applications in an Enterprise software architecture, this is not going to be the best approach to take. For enterprise-level systems, the requirements are dictated by the business and not just by a set of end users as for an individual application. Up-front work is required to ensure that the overall architecture and requirements are understood, and that planning allows for dependencies between systems to be addressed during the longer software development lifecycle that will be involved. This does not remove the need for good measures of progress and assurances of quality, which can be seen from an Agile approach, but the scale of the project must be borne in mind.

It is for this reason that project management disciplines such as Prince2, and quality control standards such as ISO-9001 have been defined. These are often maligned as being overly heavyweight, but the reaction to use lightweight approaches instead does not often resolve the problems of poor software development and delivery. A lot of failed projects are hampered by understanding of requirements, communication difficulties and false reports of progress. Agile doesn’t fix that on a large scale, although that is essentially what it does address on smaller projects.

Stepping aside from the Agile/non-Agile thoughts, I would tender that for Enterprise-level programmes, the risks of failure are perhaps equal to that of smaller and mid-sized projects, in percentage terms. However, if those odds result in an overrun or under-delivery, then the sheer scale of an enterprise software project will mean that the costs incurred are much higher, the failings more visible, with the lack of delivery being more meaningful to a larger area of the business.

Given these (personal) views, it would seem that there is more risk is using lightweight methodologies for Enterprise-level software development as a whole, whilst still permitting their use within a programme for clearly-scoped or isolated areas of functionality. Would you seriously want to consider an informal approach to developing the software applications that are supposed to drive through large-scale business change in your enterprise? Would you not want to know what you are aiming to deliver before committing to developing chunks of that software?

Another consideration is that, if you wish to move on your whole software architecture to a new platform of integrated components, such as for many SOA initiatives, perhaps the stronger thoughts should go into a phased approach. Doing ‘big bang’ software development and delivery is always an all-or-nothing commitment. Instead, aim to develop and deliver incremental functionality, integrating new with old as you go. This may seem more expensive, due to the extra integration overhead, but this allows better management of costs, reduced risks and also – as for Agile approaches – allows you to use that software from each project phase in live operation if you wish.

To sum up, I feel that software development should indeed be performed in an ‘agile’ approach regardless of scale, but that when looking to delivery Enteprise-level solutions, there is a need for Enterprise-level methodologies and programme management, which is not what gets offered by Agile approaches. I also feel that this is just what Alistair Cockburn presses for in his book; he is an Agile proponent, but he clearly recognises the need for scale to be an input to the choice of approach taken to software delivery.