Behavior Driven Development reaches Norfolk

It doesn’t always have to be a huge gig at a big plush venue that draws good speakers and skilled attendees curious to learn.

Quite often, user groups and locally-run events can be such rich sources of education and inspiration. I found Agile East Anglia to be one such group. Started by Paul Grenyer (@pjgrenyer) their deep dive sessions on aspects of Agile has been excellent. These have included Agile User Stories, Dialogue Sheets and, most recently, Behavior Driven Development with speaker  Liz Keogh (Twitter @lunivore ).

Developers from IT teams in huge corporates and developers new start-ups have been rubbing shoulders at this group which has all helped to add colour to the subject area under discussion. Content is pragmatic, realistic and nobody is looking to make a big name for themselves; all attendees have gone home the richer by learning from each other. How refreshing.

It’s good to learn that other regional groups have twigged that there is lots of good stuff happening, and a decision has been taken to merge Agile East Anglia with Norwich StartUps and Norwich Developers Community to form a group with wider reach and deeper appeal. The new group has been branded SyncNorwich (Twitter @SyncNorwich) and the inaugrual event is scheduled to take place in Norwich on 05 July 2012. Details and registration : http://www.meetup.com/SyncNorwich/events/68577412/

Meanwhile, Smart421 has engaged itself because of its hunger to fuel its deep use of Agile on numerous engagements for several enterprise Customers. For those with an interest in learning about our Smart Agile Development Process (SADP) please check out our Agile page.

Oh, and you’ll also discover a neat Android app available free for download.

 

Agile East Anglia – User Stories Explained…

On Monday night I attended my first Agile East Anglia meeting at the Assembly House in Norwich (hosted by Paul Grenyer a Smart421 associate and sponsored by Smart421) where Rachel Davies from Industrial Logic Europe Limited was guest speaker talking about user stories within agile deliveries and the benefits they bring.

Having worked on numerous programmes that have adopted and used agile delivery methodologies and practices (mainly from SCRUM and XP) and have used user stories (“As an X I want to Y so that Z”) to capture the  wants and needs of the programme I have found these to be very successful and effective so was keen to learn more; as seemed to be the case of many others with all of the allocated spaces being booked and an extra 2 people turning up for the event despite the blast of Siberian weather that has hit us over the weekend.

Rachel delivered a very comprehensive presentation based on her experiences gained over the last 11 years of using agile methodologies and provided a great introduction on capturing and using user stories effectively that went down well with the audience, who in the majority were fairly new to agile principles; she even had us all working in pairs defining and capturing our own fictitious user stories and defining our own acceptance criteria, which I think  all the attendees appreciated.

The time flew by which meant that I wasn’t able to get to cover all of the items that I wanted to such as running effective planning poker sessions, software tools available to support user story capture and management of the delivery of the user stories. I am hoping though that there will be future sessions run by the Agile East Anglia group to cover these items and more aspects and principles within the Agile delivery methodologies.

 

Gartner Local Briefing – Hyperconnected enterprises – aspiration or desperation?

Leading analyst Brian Burke talked persuasively about Hyperconnected enterprises in his presentation “Return-to-Growth Strategy: Architecting the Next-Wave Business Model” at the latest Gartner local briefing on Enterprise Architecture in London this week

Gartner’s view is that organisations are becoming more horizontally structured and more interconnected with other organisations and with their customers. To such an extent that the hyperconnected organisation will be unable to deliver without its connections.  I suspect that this is part of the continual evolution of businesses such as adoption of Just-In-Time and the exploitation of e-commerce.  Is the vertically integrated organisation a dinosaur?  Are there any left? Some of Smart421’s clients are actively pursuing strategies of horizontal integration and removing their vertical integration.

It all makes sense; changes in the environment create opportunities to be exploited, so it is natural that the Internet and fast reliable and cheap communications will be fuelling new ways of not only doing, but creating businesses.  The explosion in software services and cloud computing, which is just around the corner will accelerate this trend.  In fact it may well be that these two developments will drive much of the growth in the next economic cycle.

Brian Burke was also talking about “emergent strategies”, that is those strategies that happen outside of the corporate strategies – strategy on the fly if you like.  Gartner are suggesting that the emergent strategies are becoming more important, diminishing the effect of the more ponderous corporate strategies.  This is a wake up call to organisations with centrist attitudes.  The more distributed the organisation, the more distributed the strategies and governance need to be.  The British found that out some 200 years ago, a lesson learnt from the colonies.

What EA needs is a strategy to deal with and adopt the emergent on-the-fly strategies.

Luckily for me ticking off all the new and emerging ideas from Brian’s presentation, I found that Smart421’s EA proposition and in particular the “Sustainable EA” stance ticks all of the boxes for the hyperconnected organisation complete with an agile strategy to deal with on-the-fly strategies.

Hyperconnected organisations also reminded me of a project that I first heard about from a colleague at Smart421 on the subject of Linked Data. The project is being run out of the University of Southampton. What is particularly interesting is that Messrs Berners-Lee and Shadbolt are looking very carefully at this whole area.

 

Natural language and test driven web development

I spent a really interesting hour the other day with a colleague of mine, Steve Cresswell, going through his recent work to integrate natural language capabilities into a popular open source web test tool.

He’s been integrating JBehave with Selenium.

JBehave is a framework for Behaviour-Driven Development, and takes test-driven development (TDD) for agile to the next logical level, where instead of your business reps looking at your JUnit test with you (and hoping that the comments that explain them are up to date and consistent with the code!), you create and then execute your tests in the ultimate natural format – in English.

A simple test scenario might look like this…

Given I am logged in
When I enter an order for 10 books at a price of £2.10
Then I should see an order confirmation

Selenium is widely known in the agile community – the killer features of the Selenium IDE are that there’s very little barrier to entry for QA’s and Product Owners (customers) and unlike some of the alternatives it runs in the real browser rather than a partially simulated environment. What Steve has done is to put Selenium and JBehave together in a way that the authors hadn’t expected (i.e. having Selenium “drive” JBehave, rather than JBehave drive selenium) using AJAX + JSON to make the process more seamless…and then created a simple web app which dynamically discovers the JBehave’s “given” steps and lets you invoke them.

It’s a fascinating area and one I’d like to spend more time getting into – but heh, you can’t cover it all. But I love the idea of sitting down with your customer and creating definitions of success in English language, then building the app and those tests become your regression test suite – it takes away another barrier between the developer and customer worlds which can only mean better, quicker deliveries.

 

How does Agile work for you?

A nimble climber

I managed to make time to attend a talk on Agile Development and it was a pleasant surprise to hear an Agile practitioner speaking from first hand experience, advising to tread carefully when implementing Agile (perhaps I paraphrase a little aggressively) [see http://eastanglia.bcs.org/, "Agile Development. What must go right, what can go wrong (and what you can do about it)" by Giovanni Asproni]. The presenter was not so much suggesting any difficulty with Agile, per se, but cautioning, rather, that making any change to the way people work can be challenging, and it is something that requires careful consideration. Consideration, not just of the change itself and the team that will work it, but also of how it fits into the wider business context.

So – introduce stand ups, user stories and continuous integration after careful consideration (there are many more practices, of course); don’t introduce them all at once; and don’t introduce them just because an Agile Coach says you have to. (I maybe made up the last one.)

This is great advice: no matter how important software delivery might be in a business, it is rarely the only part. It resonates with me, and perhaps others of my ilk. I like the idea of being able to be Agile, without having to – checkbox-style – implement a prescribed set of practices, yet having the flexibility to implement those that make sense.

Yet on reflection, it leaves me feeling a little lost. I’ve found that Agile works when driving a pure software delivery: the tight feedback loop Agile offers is an incredible sight to behold. (At least, it is for compsci grads like me that grew up with university professors teaching the latest Waterfall technology has to offer.)

But scaling out beyond the development phase of a project to encompass analysis, design, integration, etc., I have found Agile to be a very challenging proposition. For sure, it “works”. But it can be hard – very hard – work. In an environment where your software makes up just one system in the picture, where other system changes are not using Agile, and where your business representative perhaps sits the other side of a contract and customer project management team, the need to consider the context the project operates in goes without saying, but more guidance is desperately needed: I wonder if checkbox-style templates are actually called for.

How does Agile “butt up” against those traditional aspects of project delivery such as requirements capture, integration and acceptance testing? What does Agile have to say about subcontracted deliverables and how should Agile be used effectively in a bid scenario? What aspects are compatible, and what are not? Does, or should, Agile ever form part of a larger Waterfall-style (or Prince 2, or …) project? These are questions that take Agile far beyond pure software delivery. But Agile is being adopted by many a large corporation, and not just for software development: whether it is ideal or not, those checkbox-style prescriptive templates will come out.

So how does Agile scale out? I don’t think we know. Many of us will get tied up in the checklist bureaucracy, and some of us will get tied up inventing it. As an Agile community, we need to start talking more about how Agile interacts with the world outside, and what we want those checklists to look like. If we don’t talk about it, those that “just use” Agile might reasonably expect we’ve talked about it and solved it. Unless I’m mistaken, that is not yet the case.

 

Aggressive agile coaches

Of the agile coaches I’ve encountered so far, the general pattern of behaviour seems to be one of displaying aggressive, almost bullying tendencies. This has been quite a surprise to me, as you’d expect them to be the very model of the agile ethos made flesh – respectful, co-operative, managing timescale pressures through prioritisation and tuning team velocities and the like.

Being a firm believer in the inherent goodness of human behaviour – i.e. in general people are pleasant, what to do a good job, want to be liked and so on – what is is about agile projects that makes people behave in a manner completely at odds with the philosophy that they are employed to uphold?

For me I think it just all comes down a few things:

• It’s not clear what experience/evidence you need to become an agile coach – but as with many things in our industry (and probably lots of others – plumbers maybe?) you have to wonder about what evidence of your skills you need to get a gig as an agile coach. Also, the skills that make a really good agile coach are quite intangible and so are difficult to measure at interview. Good references and evidence of experience can compensate for this during the recruitment process – but how many people explain the bad projects on their CV ?
• I suspect that most coaches come through the development route, i.e. lead developers who can’t resist sticking their head up when it’s all going wrong around them (which is not a bad trait). In my experience this can lead to an over-emphasis on the developmental aspects of an agile project as these are ‘home turf’. Important as they are, if the business involvement isn’t right, then your sophisticated automated testing isn’t going to rescue you.
• Some companies say they want to embrace agile methods – but organisationally they just can’t make the commitment required (and even worse think that they are), so the agile coach is already done for before he/she starts.
• All the above conspires to create a great deal of pressure on the coach – and then what does he/she do? Well, it’s time for one of two classic responses – either fight or flight – and seeing as they are being paid to deliver a project flight is not usually an option! Hence it’s time for a scrap…

So – take a moment to pity the agile coach – they are often a victim of circumstance

 

Agile and Enterprise Software Development

Having recently re-read Alistair Cockburn’s book on Agile Software Development, I thought I’d repeat some of the concepts that he describes and try to expand on how those may apply to the development of Enterprise-level software, compared to Application-level software.

There is a very interesting section describing concepts for the design of a methodology for software development. Within this, the choice of a suitable approach is based on a range of criteria, including the following:

• Methodology Size – the number of control elements in the approach
• Ceremony – a level of precision and tolerance applied by the approach; greater ceremony corresponds to tighter controls
• Methodology Weight – the product of size and ceremony, for comparison purposes
• Problem Size – the number of elements in the problem domain and their inherent cross-complexity
• Project Size – the number of people whose efforts need to be coordinated
• System Criticality – the importance of the system, or the level of damage that may by caused by any undetected faults; Alistair uses a scale as follows: loss of comfort, loss of discretionary money, loss of irreplaceable money, loss of life

Other criteria listed in his book are precision, accuracy, relevance, tolerance, visibility, scale and stability. This last one is to determine how likely things are to change, explained with a suitable statement (‘if I were to ask the same questions today and in two weeks, how likely would I be to get the same answers?’)

Those I have included in a bullet list above are perhaps most useful to classify the different types of methodologies, with the others being useful to expand on the detail of any particular approach.

The most commonly-used Agile methodologies, such as XP and Scrum, are inherently lightweight, having minimal control elements and low ceremony, with them being most suited to smaller problems and projects.

Alistair includes an interesting table showing how projects can be characterised by communication load, criticality and priorities. Discussions can then position which type of methodology should, or should not, be used for a project under consideration. Those projects with smaller teams, changing requirements, flexibility in delivery, low risk and small scale are seen at one end of the scale, whilst massive programmes and projects needing strict controls against high risk (financial or life implications) will appear at the upper end of the scale.

In many cases, development of a new piece of standalone application software can benefit from the advantages of an Agile methodology. In this, early visibility of functional software is really useful, allowing changing ideas of the purpose and features of the application to be changed as development progresses. That has to be the true home of Agile software development, backed-up by approaches such as Test-Driven Development to improve the quality of software from ‘day one’.

However, it would seem that for the development of multiple integrated applications in an Enterprise software architecture, this is not going to be the best approach to take. For enterprise-level systems, the requirements are dictated by the business and not just by a set of end users as for an individual application. Up-front work is required to ensure that the overall architecture and requirements are understood, and that planning allows for dependencies between systems to be addressed during the longer software development lifecycle that will be involved. This does not remove the need for good measures of progress and assurances of quality, which can be seen from an Agile approach, but the scale of the project must be borne in mind.

It is for this reason that project management disciplines such as Prince2, and quality control standards such as ISO-9001 have been defined. These are often maligned as being overly heavyweight, but the reaction to use lightweight approaches instead does not often resolve the problems of poor software development and delivery. A lot of failed projects are hampered by understanding of requirements, communication difficulties and false reports of progress. Agile doesn’t fix that on a large scale, although that is essentially what it does address on smaller projects.

Stepping aside from the Agile/non-Agile thoughts, I would tender that for Enterprise-level programmes, the risks of failure are perhaps equal to that of smaller and mid-sized projects, in percentage terms. However, if those odds result in an overrun or under-delivery, then the sheer scale of an enterprise software project will mean that the costs incurred are much higher, the failings more visible, with the lack of delivery being more meaningful to a larger area of the business.

Given these (personal) views, it would seem that there is more risk is using lightweight methodologies for Enterprise-level software development as a whole, whilst still permitting their use within a programme for clearly-scoped or isolated areas of functionality. Would you seriously want to consider an informal approach to developing the software applications that are supposed to drive through large-scale business change in your enterprise? Would you not want to know what you are aiming to deliver before committing to developing chunks of that software?

Another consideration is that, if you wish to move on your whole software architecture to a new platform of integrated components, such as for many SOA initiatives, perhaps the stronger thoughts should go into a phased approach. Doing ‘big bang’ software development and delivery is always an all-or-nothing commitment. Instead, aim to develop and deliver incremental functionality, integrating new with old as you go. This may seem more expensive, due to the extra integration overhead, but this allows better management of costs, reduced risks and also – as for Agile approaches – allows you to use that software from each project phase in live operation if you wish.

To sum up, I feel that software development should indeed be performed in an ‘agile’ approach regardless of scale, but that when looking to delivery Enteprise-level solutions, there is a need for Enterprise-level methodologies and programme management, which is not what gets offered by Agile approaches. I also feel that this is just what Alistair Cockburn presses for in his book; he is an Agile proponent, but he clearly recognises the need for scale to be an input to the choice of approach taken to software delivery.

 

Is Agile a Swear-word?

I’m starting to wonder if mention of the word ‘Agile’, relating to software development, is already starting to be seen as some sort of swear-word. In just the same way as ‘Waterfall’ is frowned on, even demonised.

Over my none-too-short career, I have worked in many different environments, using many different software development processes and quality standards. Those include military spec systems, banking and financial applications, telecommunications, pharmaceutical regulatory systems and assorted other application and integration projects. As a result, I have had to work on software with different requirements based on scale and rigour (consider BS5750, ISO9001 and FDA rules, for instance).

At Smart421, we have ISO accreditation for our software development processes as well as our ITIL/ISO-20000 service management activities. Being systems integrators, we will use whichever project management process is most suitable, or that which is requested by our clients. In this regard, our Prince2-based project management approach is our default choice, scaled to meet the needs of each particular project. We have also delivered projects using Rational Unified Process (RUP) and Scrum, as well as developing with other Agile software development approaches.

Given that expertise (mine and Smart421′s), it is quite clear that no one project development process is going to be correct for all software projects. Returning to my initial comment, it seems to me that ‘Agile’ may already be suffering from too much adoption on projects where it is not entirely suitable. This results in poor delivery and of customer expectations not being met, which is just the sort of problem that ‘Waterfall’ projects have been accused of over many years.

I’m not aiming to attack Agile, nor defend Waterfall, but just want to raise the issue that both have their merits and that both have a number of failures (high or low profile). The natural ground for Agile projects is for smaller scale developments, although that does not preclude it being used for large scale deliverables, but the level of rigour must be increased to allow for this. Not to say that Waterfall is the answer to delivering large projects, but it tends to bring the associated rigour (documentation, whether seen as overhead or not) needed for such systems.

Of course, Waterfall has earned its criticism – often on very large scale, large budget, failures. Agile may be lucky in that its failures may be on smaller scale, smaller projects. A further benefit is that if approached properly, even failed projects will (should) delivery something of value. If an Agile project fails to do that, it doesn’t deserve to be called Agile either.

Recall the Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more

Then read Alistair Cockburn’s book ‘Agile Software Development’ and don’t just look for the worked example on XP, but read the sentiments and meaning in there. He advocates greater controls and levels of artefacts for larger projects, based on not only the scale of the problem, but on the importance of the solution. For a life-critical system, there is a need for much more rigour than on a discretionary, nice-to-have system. All quite obvious really, but something that a number of Agile proponents seem to miss.

I intend to add a further blog article about the tension between Agile processes and enterprise level, or SOA, software, to expand on my views as to how these may or may not fit together.

In the mean time, I’d like to hope that those Agile converts don’t fail to see the wood for the trees, and that not too many projects lead to failure through inappropriate choice and use of such software development processes.

 

Agile methods and BPM

Here’s a question for you – do agile methods and BPM go hand-in-hand? I heard the other day of a project adopting an agile approach to developing automated complex back-office business processes – and at risk of falling into the trap of delivering each use case with a heavy focus on the presentation layer but not really designing/thinking about (and presumably refactoring) the business process layer. Most important is refactoring fledgling process functionality out of the presentation layer into the BPM layer.

I’m sure it can be done – but I suspect that the chances of success are significantly lower using agile methods than for a more classic pure web development. Thoughts anyone?