Agile is not a panacea.

Agile can be a powerful tool in software development but trying to force agile when the underlying enterprise architecture and design are not well thought out leads to a fragmented and disjointed system that constantly needs refactoring and re-architecting to handle major requirements.

Consider the following scenario:

Product A is based on standalone messaging modules that communicate with each other using CORBA or any proprietary messaging protocol. This leads to a situation where the product can't communicate across firewalls. The product management team wants development to re-architect this so that it can work across firewalls and while they are at it, allow the messaging modules to communicate using standard protocols so that new messaging modules can be dynamically added to the system.

Analysis:

These kinds of disruptive changes can be handled by Agile methodologies but they will end up having an extensive planning session which basically starts to look more like the waterfall approach. If the team is hurried, then the planning may not happen correctly leading to an incomplete architecture and high level design.

The agile methodology cannot pick to convert a select few messaging modules because the entire set has to work in a customer environment. This might result in creating multiple sprints( or iterations) that together will eventually cover the conversion process. During these 'sprints' the development process will end up resembling an iterative-waterfall model. 

QA on the other hand, can't fully make use of the sprints because the nightly builds they get from Dev don't completely work and are not expected to work(yet). Some may argue that this violates Agile, that the build should work but let's face it; this is not realistic. There are countless real world situations where the builds don't work while the project wide refactoring is going on.

Also we need to consider cases where software development is outsourced. You do need traditional requirements management practices to ensure that these requirements are well understood and can be implemented in the time estimated. A lot of money and reputation rests on these assumptions, so trying to do away with proper requirements management by saying that this is not agile is wrong. We need to implement requirements management process where we define how to deal with changes in requirements( we can guarantee this will happen) and what the contingency plan is.

So, I think that Agile is not a panacea, it's not going to magically solve the software development problems we are having. In fact nothing or no methodology is going to solve all the problems. Software teams have successfuly modified the waterfall model to do iterative development and added flexibility in requirements management. Not that the waterfall model is perfect but arguing that it just fails or is always prone to disaster is wrong. Here are a few things that successful team and projects have:

• Smart(er) teams have always combined processes and methodologies from multiple SDLC models to achieve the best possible result.
  
• Smart(er) teams always use common sense.
  
• Smart(er) teams have management that adapts well to requirement changes and keeps things flexible.
  
• Smart(er) teams have developers that get to focus on their work at hand and are accountable on a day by day basis.
  
• Smart(er) teams have architects that define the overall architecture, lay down the high level design for the entire release and are not restricted to break down architecture into "days". By no means this implies that Architecting is an endless and long task; actually on the contrary. But trying to box vision, thought and architecture into a "day's worth of work" is not going to help.
  

The ?(project budget) dollar questions are: 

Where does Agile apply? When exactly do I use it and how?

Agile is most applicable in situations where the architecture and design are more or less stable and where the systemic and disruptive changes have already been accounted for. There you go. It's out of my system now! Agile should be used as a "coding" methodology. Which means that the Enterprise Architecture has to exist ( it could be minimal), the product "main"(don't have to capture everything) requirements have been defined, the data architecture and the application architecture have been at least minimally defined.


Here are a few examples where agile fits:

• Where customer needs change and reordering of "small" features is necessary.
• Small development Projects
• Developing Small Features
• Adding Enhancements to an existing features.
• Bug Fixes
• Complete or proper documentation is not mandatory.

So to summarize, I advocate a hybrid approach:

• Where we don't try to be 100% Agile compliant and criticize every little deviation as being "not agile". 
• Where we finish the overall architectural model of the product or application and have a decent idea of the software development plan before we embark on strict agile practices. 
• Where the requirements management is flexible and architectural requirements are not forced-in midway into the release. We use traditional architecture and design methodologies here sprinkled with heavy doses of common sense. 
• Where the developer can be made accountable by following an Agile approach and making sure we have visibility into the day to day functioning. Too often development becomes a black box and I think this is where Agile is great. It could make the process more transparent and allow Management to peek into the progress on a day to day basis.

The hybrid approach I listed above is based on my own experiences but below are a few links that talk about a Hybrid approach to Software Development. These give me a reason to believe that I am not prejudiced or lethargic and that Agile doesn't have to be adopted "as-is". It can be modified and adapted to suit business goals. Phew...

Agile vs Waterfall approach. How about both?

Software Developers Are More Agile, But Not Entirely So
Mozilla Takes Hybrid Approach to Agile Software Development

The opinions and statements in this communication are my own and do not necessarily reflect the opinions or policies of CA.

SCA(Service Component Architecture)- Part 1

This is the part 1 in the series of articles that will cover the first three sections in the following aspects of SCA(Service Component Architecture):

• What is SCA?
• Is SCA better than JBI?
• When do I use SCA?
• How do I create an SCA Component/Composite?
• How do I consume SCA?

What is SCA?
SCA can be plainly thought of as a way to create components and then assemble them so that they can work together. The Components that make up an application, can be built in Java or any one of the programming languages supported by SCA. A different technology, for instance, BPEL can be used as well.


Component   
The components vary in complexity, ranging from a simple java classes to ones which have multiple components including local or distributed Java classes, BPEL components etc. SCA defined a logical construct called composites which is nothing but an assembly of components. An application could consist of one composite or multiple. The components could be implemented using different languages or technologies if so desired. However, the components themselves have a few fundamental features:

• Business logic exposed as Services. A Service is nothing but a set of operations that be invoked by a client.For instance a Java component might use an interface to describe the service.
• Consuming other services as References. 
• Property allows a component to read configuration or other initialization data.
• A binding can be thought of as a protocol to communicate with the service or reference and there can be multiple bindings per service/reference.

                                                           Figure 1: A component
SDO
SCA components can access data using SDO along with JDBC or JPA from a database. SDO is optional and can be thought of as the "DAO" pattern that Java defines. SDO might be the subject of another post later on.


Composite


An SCA composite is an ".composite" file which uses an XML-based format called the Service Component Definition Language (SCDL) to describe the components and their relationship.Example:




<composite name="myComposite">
   <component name="abc">
    ...
   </component>
   <component name="xyz">
      ...
   </component>
</composite>

                                                 Figure 2: A Composite




Domains
It's tough to describe domains in a way that most of us understand the word domain. Domains can be thought of as multiple SCA runtimes installed in local or distributed fashion. Since SCA doesn't define how distributed domains communicate with each other, its tough to describe what truly makes up a domain and how to map the definition of a composite to different domains? I may have not understood the specs correctly but t's probably not possible in SCA to have a composite cross multiple domains.  To me this is a significant limitation that I am hoping somebody clarifies.


(Maybe OSGI remote services can help here but that is the subject of a future post).

                                                Figure 3: An SCA Domain


Is SCA better than JBI?
It's like comparing apples to oranges ( Sorry, it's always fun to say this :-) ). To put it very succinctly JBI is what middleware vendors can use to provide or extend the runtime. SCA is what developers and assemblers can use to develop and deploy services ( as components and composites). So there is some overlap but not much. To me they are complementary and probably debated a lot because of Sun versus the "others" focus. Sun was focusing ( not sure if they still do) on JBI and the "others" were focusing on SCA. Probably what Sun is proposing has more to do with the middleware vendors themselves which the "others" are trying to avoid by focusing on the programming and component model instead of how the runtime is expected to behave.

   JBI would be more applicable for ESB vendors than component developers but I still think JBI has a role to play in Enterprise architectures because we do need more than a "point-to-point" model which SCA proposes. What if I have an SCA component that needs to broadcast a message(or event) which needs to be transformed differently for each of the target endpoints? It would be interesting to find out, how would this be implemented in SCA? Maybe we will have some "component" that either intercepts the point to point communication( is there such a thing in SCA?) or the message is actually directed to this mediator  (this breaks loose coupling?) which can then use BPEL, JMS or some other fan-out mechanism to broadcast this to the other components? Do I see an SCA+ Camel combination coming? :-)

I would hope that there is more synergy between JBI and SCA and this article does talk about one. It's from 2008, so I am not sure if this was updated or improved in recent time.

When do I use SCA?
SCA should be used to implement SOA when the organization doesn't already have any well-established SOA culture. Most often, the organization either has or tries to create a bunch of web services hoping that it would lead to SOA. Having web services is a good thing but it cannot lead to SOA unless the following criteria are met:

• Loosely coupled. A web service that depends on another web service to finish it's processing is not SOA. It cannot reliably predict if the other service is running or even SHOULD be running. Web services should be truly independant of each other and no assumptions should be made as to their availability.
• Composition of web services. Some framework or mechanism to exist that can combine, orchestrate or choreograph ( I won't go into the difference here :-) ) these services to implement business logic. Without effective coordination, having web services that are controlled by some inflexible and non-declarative piece of software is as bad as not having the web services in the first place.
• Location Transparency: Again, services should be registered and located in a way that avoids violating the loosely coupled principle. It also means that there is a single mechanism to find and locate services and it's well established.
• QoS, Governance: Without effective QoS and Governance support, web services are hard to maintain and grow. Do we know what is the impact of bringing down a particular service? Who can access a particular service? What are the Web service policy requirements and wow can a client satisfy these requirements?

SCA is a step forward in that direction as it has a bunch of specifications that can define how to create Service Oriented Business Applications ( SOBA). These specifications achieve the above SOA requirements and should be .  SCA has a programming model that allows one to use any of the supported programming languages ( C++, Java, BPEL etc).


Additionally, SCA provides the following things:

• A platform neutral service component model that spans technologies and is not dependent on a particular language.
• A client programming model to allow clients to access service components deployed in the runtime. SCA Developers can develop components using the various technologies they are familiar with, such as Enterprise Java Bean (EJBs), and use SCA to glue the components together.
• Defines a standard model for defining dependencies between components. These dependencies are defined by wiring SCA components together using references.
• Defines a standard deployment model for packaging components into a service module.
• SCA has multiple bindings ( and not just web services) including JMS, BPEL, REST etc. I read about an interesting aspect of this when someone used SCA to define a "widget" composite/component in SCA and then provided the implementation of that Widget in HTML/JS. The Widget component was using the java components as references within the HTML/JS code. JSON and ATOM binding was used to expose the java components over the wire. A very interesting example of SCA indeed! I would have to locate this resource and link it here. If any one of you can find it, please let me know. 
• An asynchronous programming model.

Caution:
An SCA runtime can implemented many different bindings including Web service, JMS, JCA, EJB etc. There is also an SCA service binding which is not defined by the spec and can be used only within a domain by an SCA runtime.The implementation of this binding type is not intended to be inter-operable. For interoperability the standardized binding types like web services should be used.

The opinions and statements in this communication are my own and do not necessarily reflect the opinions or policies of CA.

I am not against Enterprise Architecture but...

The term Enterprise Architecture has acquired a negative connotation in most enterprises. The idea that there is this one grand architecture that can solve all the business needs is not rooted in reality. How can an IT architecture solve the business problems in isolation? Is Technology good enough to solve problems without knowing what the problem is and how to measure what you solved?

I don't think so. For instance I was once involved with a project and all of us architects were showcasing the architecture to some external (theoretically unbiased) industry representatives. The demo was full of technical jargon and the latest buzzwords. The entire focus of the presentation was the architecture itself. Halfway, through the presentation, one of the industry representatives asked a simple question, "Where is business side to this? What exactly are you addressing?" There were no direct or quick answers. This was a wake up call that Architecture or Technology itself is not the solution. It is an enabler or a platform that can provide what the business is looking for now and what the business will need in a certain time from now and what the business might need in the future.

Most architects (and I have been guilty of this as well) want to sound informed and sometimes as a compulsion focus too much on the technical aspect of the architecture itself. The aspects are not required in the initial stages and the individual business components can be simplified to be architecture-agnostic. It is important to address the "might need" aspects of an enterprise but they shouldn't preclude simplicity.

An architecture design should be able to address the following things:

• First and foremost, demonstrate how the architecture design addresses what is required now and what will be required in the future. Keep discussions on what might be required in the future for possibly the second iteration of architecture design. One doesn't have to stop thinking about this but essentially not over-analyze the future.
• Which services will the architecture support(organizational unit or business unit services)?
• Which services will the architecture provide( common platform or non-functional services)? 
• Which components will the architecture support(organizational unit or business unit services or architectural )?
• What is the essential contract of a service(if any)? What are we asking a service to be? A service typically doesn't change, it can be wrapped or adapted but the basic service doesn't change. 
• What is the essential contract of a component(if any)? What are we asking a component to be? This is sometimes the hardest part because it involves changing the existing components or services in the way they exist in the enterprise. Also it is important to differentiate between components and services ( I will try to highlight this via  a separate blog post).
• Allow business requirements to be traced down to some component or module or feature. We don't want to review architecture without understanding which business problems are being solved by which features. This mapping between business requirements and the architecture design elements most probably will exist in a separate artifact.
• Resist the temptation to talk about how modules are packaged and deployed. Too many times we see an architecture design get bogged into deployment details. How a component is packaged and where it is deployed is not important at the initial stage. 

Based on the above, we will try to create a broad model that includes elements most commonly used in enterprise architectures. We will start with a simple enterprise architecture model almost resembling an applications architecture and then move to the added complexity of the enterprise. Let's assume that the architecture design is "growing" as the enterprise grows. The set of architecture guidelines that we establish will guide the designer and also allow them to "re-use" the model elements that suit their solutions. We will use elements from SOA, SCA, OSGI, JBI. We will reuse ESBs, MessageBrokers, EIP patterns. The model will aim to "assemble" these sometimes diverse elements and see if an enterprise can pick and choose the ones it needs without spending significant amount of time and money on trying to have a "grand" enterprise architecture.

So the subsequent posts will continue this thought and remember, a Software architecture is not an end to itself.

The opinions and statements in this communication are my own and do not necessarily reflect the opinions or policies of CA.

Why we keep (re)inventing a good architecture design

Thanks to the proliferation of ideas and open source tools, good integration architecture is actually a very easy thing to achieve but has been so entangled in how organizations work and how ideas are shared that it ends up becoming a fairly messy exercise. If you are still unsure, try asking two architects to agree on the details of a solution.

This article will attempt to highlight some key points in an architecture that is supposedly common across all enterprises integrations. I believe an architect’s opinions heavily influence which architecture components he/she chooses and the below are based on mine.

Bus.

This can be considered the backbone of the enterprise. Without a good distributed communication mechanism, most projects will end up investing a significant amount of time (and money) discovering how distributed components interact remotely (or locally). The difficult thing however is to select and agree on a bus.

ESBs.

A good open source ESB should do the trick but this doesn’t preclude using a proprietary one as long as the fundamental loose coupling principles are followed (but please don’t add a “wrapper on top of an existing “wrapper” to make it look loosely coupled when in fact, the underlying API or technology is not mature enough to guarantee a stable interface.

Security

I will try to avoid details at this point because this might trigger a war with the security experts. Here are my basic expectations from security:

-          How are identity and authorization service providers configured and used?

-          Who has access to the deployments and passwords? Where are these defined?

-          Can I secure the entire communication, the entire message or just one fragment of the message?

      Container (lifecycle management)

     (coming soon)


     The opinions and statements in this communication are my own and do not necessarily reflect the opinions or policies of CA.