Agile is not a panacea.

Agile can be a powerful tool in software development but trying to force agile when the underlying enterprise architecture and design are not well thought out leads to a fragmented and disjointed system that constantly needs refactoring and re-architecting to handle major requirements.

Consider the following scenario:

Product A is based on standalone messaging modules that communicate with each other using CORBA or any proprietary messaging protocol. This leads to a situation where the product can't communicate across firewalls. The product management team wants development to re-architect this so that it can work across firewalls and while they are at it, allow the messaging modules to communicate using standard protocols so that new messaging modules can be dynamically added to the system.

Analysis:

These kinds of disruptive changes can be handled by Agile methodologies but they will end up having an extensive planning session which basically starts to look more like the waterfall approach. If the team is hurried, then the planning may not happen correctly leading to an incomplete architecture and high level design.

The agile methodology cannot pick to convert a select few messaging modules because the entire set has to work in a customer environment. This might result in creating multiple sprints( or iterations) that together will eventually cover the conversion process. During these 'sprints' the development process will end up resembling an iterative-waterfall model. 

QA on the other hand, can't fully make use of the sprints because the nightly builds they get from Dev don't completely work and are not expected to work(yet). Some may argue that this violates Agile, that the build should work but let's face it; this is not realistic. There are countless real world situations where the builds don't work while the project wide refactoring is going on.

Also we need to consider cases where software development is outsourced. You do need traditional requirements management practices to ensure that these requirements are well understood and can be implemented in the time estimated. A lot of money and reputation rests on these assumptions, so trying to do away with proper requirements management by saying that this is not agile is wrong. We need to implement requirements management process where we define how to deal with changes in requirements( we can guarantee this will happen) and what the contingency plan is.

So, I think that Agile is not a panacea, it's not going to magically solve the software development problems we are having. In fact nothing or no methodology is going to solve all the problems. Software teams have successfuly modified the waterfall model to do iterative development and added flexibility in requirements management. Not that the waterfall model is perfect but arguing that it just fails or is always prone to disaster is wrong. Here are a few things that successful team and projects have:

• Smart(er) teams have always combined processes and methodologies from multiple SDLC models to achieve the best possible result.
  
• Smart(er) teams always use common sense.
  
• Smart(er) teams have management that adapts well to requirement changes and keeps things flexible.
  
• Smart(er) teams have developers that get to focus on their work at hand and are accountable on a day by day basis.
  
• Smart(er) teams have architects that define the overall architecture, lay down the high level design for the entire release and are not restricted to break down architecture into "days". By no means this implies that Architecting is an endless and long task; actually on the contrary. But trying to box vision, thought and architecture into a "day's worth of work" is not going to help.
  

The ?(project budget) dollar questions are: 

Where does Agile apply? When exactly do I use it and how?

Agile is most applicable in situations where the architecture and design are more or less stable and where the systemic and disruptive changes have already been accounted for. There you go. It's out of my system now! Agile should be used as a "coding" methodology. Which means that the Enterprise Architecture has to exist ( it could be minimal), the product "main"(don't have to capture everything) requirements have been defined, the data architecture and the application architecture have been at least minimally defined.


Here are a few examples where agile fits:

• Where customer needs change and reordering of "small" features is necessary.
• Small development Projects
• Developing Small Features
• Adding Enhancements to an existing features.
• Bug Fixes
• Complete or proper documentation is not mandatory.

So to summarize, I advocate a hybrid approach:

• Where we don't try to be 100% Agile compliant and criticize every little deviation as being "not agile". 
• Where we finish the overall architectural model of the product or application and have a decent idea of the software development plan before we embark on strict agile practices. 
• Where the requirements management is flexible and architectural requirements are not forced-in midway into the release. We use traditional architecture and design methodologies here sprinkled with heavy doses of common sense. 
• Where the developer can be made accountable by following an Agile approach and making sure we have visibility into the day to day functioning. Too often development becomes a black box and I think this is where Agile is great. It could make the process more transparent and allow Management to peek into the progress on a day to day basis.

The hybrid approach I listed above is based on my own experiences but below are a few links that talk about a Hybrid approach to Software Development. These give me a reason to believe that I am not prejudiced or lethargic and that Agile doesn't have to be adopted "as-is". It can be modified and adapted to suit business goals. Phew...

Agile vs Waterfall approach. How about both?

Software Developers Are More Agile, But Not Entirely So
Mozilla Takes Hybrid Approach to Agile Software Development

The opinions and statements in this communication are my own and do not necessarily reflect the opinions or policies of CA.