Reality check for Agile projects- collocation vs virtual collocation

We all have an ideal view of the world surrounding us. That is OK as long as we are aware of the real world and know how to rationally deal with it. . We as agile practitioners also have our own view of an ideal agile world. I will talk about some of the "aspects" of this ideal agile world in a series of blogs. This is first in the series. Today the topic is "Collocation vs virtual collocation."

We all agree that face-to-face communication is the best. So much so that it has an explicit place in the Agile principles. However, we see growing number of projects have distributed teams for many reasons- outsourcing and offshoring, telecommute, multiple offices, lack of local talents etc. I think it is time for us to embrace the reality (as Agile was born from the realization of embracing the reality vs fighting it like the waterfall camp did) and update the Agile principle to say

"The most efficient and effective method of conveying information to and within a development team is face-to-face conversation in person or virtually."

Outsourcing and offshoring has become mainstream. It is here to stay for foreseeable future. It is elegantly explained by Thomas Friedman in his famous "World is flat" book. If you haven't read this book yet and want to understand the dynamics of today's connected world, I would highly recommend it.

Having a team distributed across continents is not any more a "tactical step," it is a "strategic need." Think where would Jason Fried and 37signals be without DHH, if it were not for "virtual collocation." There are many such success stories. If you are not open to working with the right talents irrespective of their location, you are at a great disadvantage compared with your competitors. Let me be a little blunt here- forget about competing.

In an increasingly crowded world, it is a growing trend for people telecommuting. Why would we require people to come together at a designated physical location everyday when a significant time and energy is wasted on the road? Why not we enjoy living in home town and being able to spend more time with family, yet have the opportunity to work on projects that could benefit from my skills and experiences?

Next generation (generation Y) are growing up in a virtual world (Facebook+SMS+Twitter+blog+iPhone). Virtual collocation is in their gene. They are forging lifetime friendship in this virtual world, coming together to support a cause, interacting on a daily basis as if they are in the same room. When they enter the workforce, they would not know anything different. In fact, they would prefer to work in companies where it is part of the culture.

So, agile practices should evolve to embrace virtual collocation rather than ignoring it, even worse resisting it. In fact, Agile is not at odds with virtual collocation. It is rather a necessity for distributed projects. There are many examples of successful adoption of Agile on distributed projects- large and small.

Yes, it is not easy to get it right. We may have to work a little harder to get it right. I would argue that getting communication right in a collocated environment isn't a cake walk either. Right communication is more than just being collocated.

I make a prediction here- in a decade or so most projects will have distributed teams. Collocation will not be a topic of conversation any more. 895gsubzpr

The most important software engineering practice

Continuous Integration (CI), hands down. I hope you have already heard the term if you are in anyway associated with software development. If not, I would highly, highly encourage you to read this article by Martin Fowler.

There are many practices in the Agile community, specially among XPers. Among those, TDD (test driven development), refactoring, and pair programming are other three most talked about practices aside from CI. Most of these practices have believers and non-believers.

TDD. Recently Uncle Bob preached TDD to all developers in his keynote speech at RailsConf 2009. Although I am a big fan of Uncle Bob and learned a lot from his writings over the years, I am afraid I have to side with DHH on this practice. Don't get me wrong. I like TDD, but I do not think it is a key to "clean code." Test first approach is not natural, it takes a lot of practice. Even then it may seem difficult to many developers. I am of the opinion that whether you write test first and then code or vice versa, you must write tests. But, follow what feels natural to you. That way you as a developer will have fun doing it. I agree with DHH that it is important that we have fun while writing clean codes. At the end, as a client/sponsor I do not care whether tests were written first or not, what matters is whether the software comes with comprehensive tests or not (code coverage). By the way, if you haven't seen Uncle Bob speak, you must. He is very animated speaker.

Refactoring. Although most projects/products go through refactoring at least once (if not more), it meets with resistance from all sides. Because, it comes with the connotations like low quality, incompetence, waste etc. Developers do not want to admit that their own codes need rework. The project managers do not want to ask clients to pay for something that is already done and working. The clients when hear "refactoring" becomes anxious thinking that the software that they are paying for is of low quality. The other end of the spectrum is teams can get overzealous and end up spending in this endless cycle of refactoring. So, it is difficult to implement "just enough" continuous refctoring that helps a team to maintain a sustainable forward pace on the project. When that right balance is struck, refactoring helps design to emerge/evolve as well as keep technical debt to a minimum.

Pair programing. One of the most controversial practices. I have to admit that I was a non-believer too until recently. My team has recently started doing pair programming and found it to be very helpful specially for refactoring type and complex works. It probably is not as effective for all types of work as well as every team. The most challenging aspect of adopting this practice is to justify upfront increased, however small or large, development costs and time and convince clients to pay for it. At least in the minds of clients, two persons working together on one thing at any time will look like they will be paying twice as much for the same amount of work and probably will take twice as long. It is a hard sell.

CI. It is the most accepted practice with very tangible benefits that both technical and non-technical persons can appreciate. If your team is not currently doing it, you should push for implementing CI immediately. If your team is already doing it, make sure you have the 10 elements mentioned in Martin Fowler's article. The diagram below shows the main steps involved in CI.



Most tools will generate Web reports and send email every time CI process runs. As a client, or PM, or manager, you would want to be on the distribution list for this report. Also you should learn how to read the report so that you understand what it means for your project. Even if you are not following Agile process, you can still benefit from CI.

Containing technical debt

One of the most important concepts in Software Engineering is "Technical Debt." It was first introduced by Ward Cunningham in 1992 in an experience report at the OOPSLA. If you haven't heard the term before or want to refresh your memory, I have collected articles/blogs on the topic that I thought have done a good job of explaining it.

• Here is Ward Cunningham explaining his thoughts behind why he coined the metaphor on wiki and also in a video talk.
  
• Here is a blog by Martin Fowler on the topic.
  
• Here is a personal blog by Dharmesh on the same topic.

I was at an Open Space session on technical debt at Orlando Scrum Gathering early this year. What seemed to be on everybody's mind was how to protect our projects from going bankrupt under the burden of technical debt. We all agreed that it is easy to deal with debts that we take on knowingly. The real challenge is to identify debts that creep up on us over time (like credit card debt- a little here, a little there, some missed payments, and voila...). We get caught by surprise until it is too late.

I am going to suggest a few things when put into regular practice can help you contain the debt to a minimum.

1. Low test coverage can contribute to hidden technical debt. When we are under time pressure, guess what gets cut from the budget? "testing!" It may seem innocuous to eliminate funding for "testing," it can push the project to bankruptcy when done on a regular basis. So, target 90%+ test coverage and track it through continuous integration.

2. The best "architecture" is not static, it is evolutionary that emerges over time. What is important though is "consistency." If we are always doing the same thing the same way, the chances of incurring hidden debt goes down. The consistent way of doing things needs to be a shared knowledge best captured as a list of "How Tos" and needs to be kept in synch with the evolving architecture. Do you have one such list for your project?

3. Review architecture at a regular interval to spot inconsistencies. I am sure you will find some even after you have a published "How Tos." As soon as you find them, fix them.

4. I have seen bugs are triaged and then some are deferred to be fixed in the future in favor of some "important" new features. The fallacy lies in the notion that cosmetic/minor bugs are harmless. What is harmless today can compound into a significant issue later somewhere else in the system. Personally, I always like to fix all bugs before I do any new development. However, I can understand that we may need to do it at times, but it should be "exception" than "norm."

5. Last but not least, if you must take on a debt, get a commitment to payback in full by a specified timeline. Then track and report on it like a hawk (not like the Fed or SEC).

How are you containing technical debt on your project? Care to share?

Why should you consider Ruby on Rails for your next project?

I am sure many of you are asking yourself- so, why should I use Ruby on Rails (a.k.a. RoR) for my next project? I am going to share with you why I would. It is the Rails, the application framework, that is the force behind RoR. What Rails does is constrains developers in this predefined application structure with necessary elements required to build a reasonably good enterprise/commercial strength Web-based solution. Constraints can be liberating. It is certainly true for RoR. As a result, you can expect to build an application of reasonable quality with a less experienced/young team faster than any other technology, for example, Java/.Net (assumption is the team has the same years of experience with both technologies ). The database migration tool that it provides to help with incrementally building the database alone can be reason enough to give it a serious thought. Rails' simple yet elegant plugin architecture have fostered a large active community of plugin builders. You can find many useful plugins that are needed for many applications. These plugins could speed up your development time even faster.

We all know imitation is the highest form of flattery. It is true for Rails too. Rails has inspired many copycat frameworks in many languages. Even Microsoft has taken a notice of it. Microsoft has recently published its Rails-inspired MVC framework for its .Net platform. Please remember, these Rails-inspired MVC frameworks do not give you all the things that you get from Rails to make your team hyper productive.

While RoR philosophy of "Convention over Configuration" contributes to developers' productivity, it could also be limiting for certain situations. RoR is good for new applications as well as porting existing applications where your existing databases can easily support "Single Table Inheritance" in your domain.

I have heard many expressed concerns about Ruby's performance because it is a dynamic language. It reminds me of Java's early days. We have come a long way since then. We have many high traffic RoR applications in production. Ruby is now supported on Java VM through JRuby and on .Net CLR through IronRuby. Performance should not be a concern for RoR applications that can also be built using Java or .Net. If you are not convinced, here is a very good case study on YellowPages.com that should remove all your concerns about RoR. If you are still not convinced, David Hansson, the man behind the Rails, talks about why scaling with RoR is boring...:-)

Are you considering RoR for your next project? Did you recently complete a RoR project? Please feel free to share your experience with us here.

The Dream Team of Agile Software Development

If you had to put together the best team possible for the most complex project and money is not a concern, who would you have on your team? I can tell you who would be on my "The Dream Team."

Product Owner- Mike Cohen

Scrum Master- Ken Schwaber

The Development Team

Kent Beck,
Bob Martin,
Jim Copline,
Martin Fowler,
Alistair Cockburn,
Ward Cunningham,
Jeff Sutherland,
Ron Jeffries, and
Jim Highsmith

I am making an assumption that they all still write software and will be able to work with each other...:-) Now the questions are-

• Is this team going to be the most hyper-productive team ever in the history of software industry?
• Are they going to follow Scrum? Or, are they going to come up with something new? Should we call that "Scrum But?"

Just wondering. Unfortunately, the reality is much more constrained. We need to work with a team that we can reasonably put together. The reasonable expectation is to have individuals who are average. I always wonder why people are so obsessed with hiring the best/brightest from Standford or MIT. I guess I do not fall in that category (and hence the grape is sour!)...:-) However, I sure can bet my money on my team that it can go head-to-head with any dream team in the world. I am sure so is yours.

The focus should be "The Team," not the individual. Do me a favor, when you look for a new team member, mention in your Ad- "Looking for the best team member."

What do you think? Who would be on your dream team?

Orlando Scrum Gathering 2009

I just came back from the gathering last week. It was different this time from the past gatherings in that this time the event was graced by the presence of industry heavy hitters like Ron Jeffries (Co-developer of XP), Jim Copline (The author of "Advanced C++" that helped me shape my understanding of C++ and OO early in my career), Alistair Cockburn (famous for his book "Writing Effective Use Cases" and Agile process called "Crystal"), Dr. Mark Paulk (led the work on CMM at SEI), and last but not least Gregory Balestrero (President and CEO of PMI). As you can see from the lineup, Scrum Alliance has done a great job of inviting these people and initiating an industry wide collaboration around software development process.

What to expect in the coming months? Scrum Alliance has engaged Dr. Paulk to do empirical research on Scrum. He is going to focus on Agile and Scrum practices and help the Agile community to understand what practices really contribute to the success of projects. With the inclusion of PMI, we would see more and more practices being included in the PMBOK. There is a growing trend among PMPs to adopt Scrum and Agile practices. A group of PMI members has already formed a pmiagile Yahoo group to promote agile practices within PMI. This group has been officially recognized by the PMI.

What's Scrum community grappling with? From the types of discussions I heard at the event, it seems the adoption of Scrum has come a long way. The top things now-a-days on the minds of most Agile teams are,

1. User Story vs Use Case. It was thought that User Stories are going to replace Use Cases. However, it seems people are still using Use Cases (especially on large projects) in at least some light weight format along with User Stories as they lack high level contexts. Jeff Payton's "User Story Map" technique to address this gap in User Story is becoming popular. He was at the gathering and offered sessions on it. Alistair is a vocal advocate of continuing to use Use Cases. Surprise, surprise...:-) He wrote an article on User Stories vs Use Cases for the Lean magazine (I thought I could get the article from Alistair's site or from the magazine's, but I could not).
   



2. Agile Architecture. Jeff Sutherland and Jim Coplien had a session on how architecture can help a team becoming more Agile. There is a trend among Agile teams to only rely on continuous refactoring to have a suitable/consistent architecture emerge over time. They lose sight of the importance of starting on the right foot with some upfront "architecture thinking." Jim is working on an new architectural pattern called DCI (data, context and interaction) that could set the OO developers free from the shortcomings ("algorithm" part of the stories is usually scattered among many domain entities/models and get in the way to becoming Agile) associated with MVC.
   



3. ScrumBut. Most teams can be considered ScrumBut, a termed coined by Ken Schwaber (I think) to point out how teams are not taking full advantage of Scrum practices. While the intent of ScumBut is to help team identify where they have improvement opportunities, it is creating confusion and frustration in the minds of the practitioners in that if there is such thing called "Pure Scrum" (which seems to be an illusive target). The whole confusion around the term ScrumBut (which gathered more negative, almost derogatory, connotation than it was intended for) lies in its definition based on a check list of mechanics. In fact, it was brought up in the panel discussion at the gathering that the definition of Scrum as found on the Scrum Alliance site as 3 roles, 3 rituals, and 3 tools hide the true essences of Agile captured in the Agile Manifesto. The mechanics are there to help teams get started, but they, by no means, are "sliver bullets." Each team needs to find its unique practices (customized through "inspect and adapt") that help them incrementally deliver working software that maximizes value for the customers.

Alistair nicely summed up the underpinnings of Agile process movement this way,

Self-awareness (focus of Crystal), self-discipline (focus of XP), and self-organization (focus of Scrum).

Best Practices for Amazon Web Services (Amazon Cloud)

We run our ScrumPad (A Web-based project management tool) on Amazon cloud (AWS). We have been using Amazon cloud for last year and a half. We did not have any issue with it until recently, when we found out that we had lost shell connection to our production server. Although the application was working fine, we went into a panic mode realizing that we won't be able to do anything, should we encounter any problem. Once we calmed down from our initial panic attack, we quickly searched on the net to see if anyone experienced the same issue and how they recovered from it. We did not get any hit on it. There is always first time for everything. We thought to ourselves that the solution is easy with AWS. We would just bring up another EC2 instance and point to the new instance. Right? Not so fast. We found out that we were not using elastic IP. That means we need to make DNS change, which takes up to 24 hours to propagate and may result into some data loss and/or service downtime.

This incident helped us to think through how we should be using AWS and here is what we have come up with:

1. Always use elastic IP (behaves like a static ip) to point to your EC2 instance. You can then quickly swap instances without needing to make any DNS changes.

2. Always use EBS (elastic block storage) instead of the storage that comes with an EC2 instance to store your Database data . You can then quickly reattach your data store to a new instance as well as you will not loose your data in the event of EC2 instance failure and have your data automatically replicated.

3. Spread your EC2 instances to different availability zones and regions instead of putting them all in the same availability zone (a physical data center). You can then protect your application from single data center failure.

4. Backup (incremental snap shots) your database from EBS to S3 (Simple Storage Service), which is automatically replicated across multiple availability zones.

5. Create a custom AMI (Amazon Machine Image) for your application from a pre-configured public AMI. You can then bring up a new instance quickly (in less than 15 mins) on-demand and even completely automate server provisioning.

6. Rollout your new release to a new instance(s) when you need to change your AMI (i.e., when you upgrading your DB, OS, Application Server, etc.) and then reattach your elastic ip and EBS to this new instance. You can then quickly rollback to the old instance should you encounter any problem with the new release.

Figure: How to transparently switch to a new instance

7. If budget permits, use cloud management tool like RightScale.

With a few practices in place, you can have your own virtual data center (on Amazon cloud) that was only available to large corporations.