On Monday night I attended my first Agile East Anglia meeting at the Assembly House in Norwich (hosted by Paul Grenyer a Smart421 associate and sponsored by Smart421) where Rachel Davies from Industrial Logic Europe Limited was guest speaker talking about user stories within agile deliveries and the benefits they bring.

Having worked on numerous programmes that have adopted and used agile delivery methodologies and practices (mainly from SCRUM and XP) and have used user stories (“As an X I want to Y so that Z”) to capture the  wants and needs of the programme I have found these to be very successful and effective so was keen to learn more; as seemed to be the case of many others with all of the allocated spaces being booked and an extra 2 people turning up for the event despite the blast of Siberian weather that has hit us over the weekend.

Rachel delivered a very comprehensive presentation based on her experiences gained over the last 11 years of using agile methodologies and provided a great introduction on capturing and using user stories effectively that went down well with the audience, who in the majority were fairly new to agile principles; she even had us all working in pairs defining and capturing our own fictitious user stories and defining our own acceptance criteria, which I think  all the attendees appreciated.

The time flew by which meant that I wasn’t able to get to cover all of the items that I wanted to such as running effective planning poker sessions, software tools available to support user story capture and management of the delivery of the user stories. I am hoping though that there will be future sessions run by the Agile East Anglia group to cover these items and more aspects and principles within the Agile delivery methodologies.

policeOf the agile coaches I’ve encountered so far, the general pattern of behaviour seems to be one of displaying aggressive, almost bullying tendencies. This has been quite a surprise to me, as you’d expect them to be the very model of the agile ethos made flesh – respectful, co-operative, managing timescale pressures through prioritisation and tuning team velocities and the like.

Being a firm believer in the inherent goodness of human behaviour – i.e. in general people are pleasant, what to do a good job, want to be liked and so on – what is is about agile projects that makes people behave in a manner completely at odds with the philosophy that they are employed to uphold?

For me I think it just all comes down a few things:

  • It’s not clear what experience/evidence you need to become an agile coach – but as with many things in our industry (and probably lots of others – plumbers maybe?) you have to wonder about what evidence of your skills you need to get a gig as an agile coach. Also, the skills that make a really good agile coach are quite intangible and so are difficult to measure at interview. Good references and evidence of experience can compensate for this during the recruitment process – but how many people explain the bad projects on their CV ?
  • I suspect that most coaches come through the development route, i.e. lead developers who can’t resist sticking their head up when it’s all going wrong around them (which is not a bad trait). In my experience this can lead to an over-emphasis on the developmental aspects of an agile project as these are ‘home turf’. Important as they are, if the business involvement isn’t right, then your sophisticated automated testing isn’t going to rescue you.
  • Some companies say they want to embrace agile methods – but organisationally they just can’t make the commitment required (and even worse think that they are), so the agile coach is already done for before he/she starts.
  • All the above conspires to create a great deal of pressure on the coach – and then what does he/she do? Well, it’s time for one of two classic responses – either fight or flight – and seeing as they are being paid to deliver a project flight is not usually an option! Hence it’s time for a scrap…

So – take a moment to pity the agile coach – they are often a victim of circumstance :)

Having recently re-read Alistair Cockburn’s book on Agile Software Development, I thought I’d repeat some of the concepts that he describes and try to expand on how those may apply to the development of Enterprise-level software, compared to Application-level software.

There is a very interesting section describing concepts for the design of a methodology for software development. Within this, the choice of a suitable approach is based on a range of criteria, including the following:

  • Methodology Size – the number of control elements in the approach
  • Ceremony – a level of precision and tolerance applied by the approach; greater ceremony corresponds to tighter controls
  • Methodology Weight – the product of size and ceremony, for comparison purposes
  • Problem Size – the number of elements in the problem domain and their inherent cross-complexity
  • Project Size – the number of people whose efforts need to be coordinated
  • System Criticality – the importance of the system, or the level of damage that may by caused by any undetected faults; Alistair uses a scale as follows: loss of comfort, loss of discretionary money, loss of irreplaceable money, loss of life

Other criteria listed in his book are precision, accuracy, relevance, tolerance, visibility, scale and stability. This last one is to determine how likely things are to change, explained with a suitable statement (‘if I were to ask the same questions today and in two weeks, how likely would I be to get the same answers?’)

Those I have included in a bullet list above are perhaps most useful to classify the different types of methodologies, with the others being useful to expand on the detail of any particular approach.

The most commonly-used Agile methodologies, such as XP and Scrum, are inherently lightweight, having minimal control elements and low ceremony, with them being most suited to smaller problems and projects.

Alistair includes an interesting table showing how projects can be characterised by communication load, criticality and priorities. Discussions can then position which type of methodology should, or should not, be used for a project under consideration. Those projects with smaller teams, changing requirements, flexibility in delivery, low risk and small scale are seen at one end of the scale, whilst massive programmes and projects needing strict controls against high risk (financial or life implications) will appear at the upper end of the scale.

In many cases, development of a new piece of standalone application software can benefit from the advantages of an Agile methodology. In this, early visibility of functional software is really useful, allowing changing ideas of the purpose and features of the application to be changed as development progresses. That has to be the true home of Agile software development, backed-up by approaches such as Test-Driven Development to improve the quality of software from ‘day one’.

However, it would seem that for the development of multiple integrated applications in an Enterprise software architecture, this is not going to be the best approach to take. For enterprise-level systems, the requirements are dictated by the business and not just by a set of end users as for an individual application. Up-front work is required to ensure that the overall architecture and requirements are understood, and that planning allows for dependencies between systems to be addressed during the longer software development lifecycle that will be involved. This does not remove the need for good measures of progress and assurances of quality, which can be seen from an Agile approach, but the scale of the project must be borne in mind.

It is for this reason that project management disciplines such as Prince2, and quality control standards such as ISO-9001 have been defined. These are often maligned as being overly heavyweight, but the reaction to use lightweight approaches instead does not often resolve the problems of poor software development and delivery. A lot of failed projects are hampered by understanding of requirements, communication difficulties and false reports of progress. Agile doesn’t fix that on a large scale, although that is essentially what it does address on smaller projects.

Stepping aside from the Agile/non-Agile thoughts, I would tender that for Enterprise-level programmes, the risks of failure are perhaps equal to that of smaller and mid-sized projects, in percentage terms. However, if those odds result in an overrun or under-delivery, then the sheer scale of an enterprise software project will mean that the costs incurred are much higher, the failings more visible, with the lack of delivery being more meaningful to a larger area of the business.

Given these (personal) views, it would seem that there is more risk is using lightweight methodologies for Enterprise-level software development as a whole, whilst still permitting their use within a programme for clearly-scoped or isolated areas of functionality. Would you seriously want to consider an informal approach to developing the software applications that are supposed to drive through large-scale business change in your enterprise? Would you not want to know what you are aiming to deliver before committing to developing chunks of that software?

Another consideration is that, if you wish to move on your whole software architecture to a new platform of integrated components, such as for many SOA initiatives, perhaps the stronger thoughts should go into a phased approach. Doing ‘big bang’ software development and delivery is always an all-or-nothing commitment. Instead, aim to develop and deliver incremental functionality, integrating new with old as you go. This may seem more expensive, due to the extra integration overhead, but this allows better management of costs, reduced risks and also – as for Agile approaches – allows you to use that software from each project phase in live operation if you wish.

To sum up, I feel that software development should indeed be performed in an ‘agile’ approach regardless of scale, but that when looking to delivery Enteprise-level solutions, there is a need for Enterprise-level methodologies and programme management, which is not what gets offered by Agile approaches. I also feel that this is just what Alistair Cockburn presses for in his book; he is an Agile proponent, but he clearly recognises the need for scale to be an input to the choice of approach taken to software delivery.

I’m starting to wonder if mention of the word ‘Agile’, relating to software development, is already starting to be seen as some sort of swear-word. In just the same way as ‘Waterfall’ is frowned on, even demonised.

Over my none-too-short career, I have worked in many different environments, using many different software development processes and quality standards. Those include military spec systems, banking and financial applications, telecommunications, pharmaceutical regulatory systems and assorted other application and integration projects. As a result, I have had to work on software with different requirements based on scale and rigour (consider BS5750, ISO9001 and FDA rules, for instance).

At Smart421, we have ISO accreditation for our software development processes as well as our ITIL/ISO-20000 service management activities. Being systems integrators, we will use whichever project management process is most suitable, or that which is requested by our clients. In this regard, our Prince2-based project management approach is our default choice, scaled to meet the needs of each particular project. We have also delivered projects using Rational Unified Process (RUP) and Scrum, as well as developing with other Agile software development approaches.

Given that expertise (mine and Smart421’s), it is quite clear that no one project development process is going to be correct for all software projects. Returning to my initial comment, it seems to me that ‘Agile’ may already be suffering from too much adoption on projects where it is not entirely suitable. This results in poor delivery and of customer expectations not being met, which is just the sort of problem that ‘Waterfall’ projects have been accused of over many years.

I’m not aiming to attack Agile, nor defend Waterfall, but just want to raise the issue that both have their merits and that both have a number of failures (high or low profile). The natural ground for Agile projects is for smaller scale developments, although that does not preclude it being used for large scale deliverables, but the level of rigour must be increased to allow for this. Not to say that Waterfall is the answer to delivering large projects, but it tends to bring the associated rigour (documentation, whether seen as overhead or not) needed for such systems.

Of course, Waterfall has earned its criticism – often on very large scale, large budget, failures. Agile may be lucky in that its failures may be on smaller scale, smaller projects. A further benefit is that if approached properly, even failed projects will (should) delivery something of value. If an Agile project fails to do that, it doesn’t deserve to be called Agile either.

Recall the Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more

Then read Alistair Cockburn’s book ‘Agile Software Development’ and don’t just look for the worked example on XP, but read the sentiments and meaning in there. He advocates greater controls and levels of artefacts for larger projects, based on not only the scale of the problem, but on the importance of the solution. For a life-critical system, there is a need for much more rigour than on a discretionary, nice-to-have system. All quite obvious really, but something that a number of Agile proponents seem to miss.

I intend to add a further blog article about the tension between Agile processes and enterprise level, or SOA, software, to expand on my views as to how these may or may not fit together.

In the mean time, I’d like to hope that those Agile converts don’t fail to see the wood for the trees, and that not too many projects lead to failure through inappropriate choice and use of such software development processes.


Get every new post delivered to your Inbox.

Join 1,122 other followers