Seven Tips to Running an Agile/Scrum Project within a Waterfall Organization

If you are working in a waterfall organization and would like to manage your next project using Agile/Scrum, I have compiled seven tips for you to help with that. This is based on my experience implementing Agile/Scrum on a federally funded, large multi-agency Government project for last two and a half years.

Some organizations are probably better off operating as "waterfall" organization. However, these waterfall organizations still can benefit from running some projects using Agile/Scrum without needing to force the whole organization to suddenly change. These projects can run internally as Agile while they interface with the waterfall organization through waterfall artifacts.

Here are the seven tips:

1. Continue to use the traditional artifacts to interface with the waterfall organization. Probably your waterfall organization is accustomed to seeing certain project documents, i.e., weekly status report, project plan using MS project, traditional requirements document using use cases or some other format, architecture document, issues/risk list etc. You can map the Agile artifacts to these documents and share them with the rest of the organization. This will help eliminate any objection to using Agile/Scrum on your project.

2. Choose sprint length appropriately. You do not want to go too fast and find yourself waiting for the waterfall organization to respond to your project needs. Stop and go would interrupt your project flow. On the other hand, you do not want to go too slow either. Most probably your waterfall organization is already slow enough and if you go at the same pace, you probably will unnecessarily burn $s. A sprint length of 3/4 weeks is probably a better choice.

3. Get external resource commitments prior to start of a sprint. Your project most likely depends on people (i.e., for HW&SW procurement, system administration, domain expertise etc.) who are not full-time on your project and operate in waterfall fashion to do their work. I call them “waterfall elements” of your project. You need to understand the lead time required (in your organization) to have these people available for your project when you need them.

4. Use a proxy to represent waterfall work streams to your daily Scrum and Scrum Planning. You probably have work items that will be performed by people who work in a waterfall manner. This is even more critical when you need to have certain pieces of software developed by a waterfall team. You would need a proxy who preferably is part of the waterfall team and can represent the work being done to the core project team during daily Scrum and Scrum planning meetings.

5. Create a sense of urgency and excitement within the project team and among the stakeholders. Most likely you may not have a client who proactively shows enthusiasm about project. It is especially true in Government and non-profit organizations. Lack of presence of clients in front of the project team can be demoralizing. Try and find a creative way to invite your clients/users/stakeholders to the sprint review meetings. This will create much needed urgency in your project team and will eventually create genuine excitement among the stakeholders.

6. Choose the right Scrum Master who understands your organizational politics. Most likely your organization has many ceremonies (documented and undocumented) as well as a network of influence. Scrum Master should be well versed with these long held practices and acquainted with the network of influence. Because he is going to need to leverage the political landscape to help remove impediments quickly as well as shield the team from external noises arising from the organizational politics.

7. Use retrospective to gauge the degree of self-organization of your project team. Most likely your project team does not have prior experience as a team to work in a self-organizing environment. Early in the project, you have to focus on encouraging behaviors that promote self-organization. As a Scrum master, you also have to protect the team from the rest of the organization getting in their way to becoming self-organizing team. Self-organization is the key to the success of any agile project.

If you are already running a project using Agile/Scrum in a waterfall organization, I would love to hear from you and compare notes.

7 Practices to Agile QA

If you are transitioning to Agile/Scrum and wondering how your existing QA practices need to evolve in the new world, here are 7 practices that you should consider implementing.

1. "In-cycle" QA. Traditionally QA team is a separate team and . QA testers should be integral part of the development team. They should be working very closely with the developers on stories in the same sprint/iteration. If QA testing is done in the next sprint for stories implemented in the previous (out-of-cycle), the team will always find itself doing a catch-up and end up accumulating "technical debt." As a rule of thumb, you would need 1 tester for every 3 developers. However, everybody on the team should be willing to jump in and help with QA testing. When planning, make sure to adequately account for QA activities. It is OK to take on fewer stories in the beginning to find the team rhythm before trying to increase the velocity.

2. Implement 5 Quality Gates. Traditionally QA is seen as just performing functional/system tests. But, software development is a highly integrated process. The quality of each activity- requirements, design, and coding, could impact the amount of effort needed in testing. If quality issues that could have been detected and prevented during these other activities are left for testing to catch them, there may not be enough time to fix the issues. Even worse, it may not be found until after deployment. QA needs to expand beyond just testing and should include the following QA gates.

Fig. 1: Quality Funnel

• Quality Gate#1- peer review of requirements. Make sure stories meet INVEST (independent, negotiable, valuable, estimable, small, and testable) criteria as much as possible. This could be an on-going activity as part of pre-planning/product backlog grooming and not tied to a specific sprint.
• Quality Gate#2- peer review of design. Make sure the design is in-line with the architecture guideline/style and appropriate alternatives are considered.
• Quality Gate#3- peer review of codes. Make sure appropriate unit tests are written, coding standards are followed, and above all ensure codes follows design.
• Quality Gate#4- continuous integration. Make sure code is being integrated and automated tests are run on a continuous basis and any issues that surfaces are addressed immediately.
• Quality Gate#5- automated functional tests. Make sure QA testers are testing using automated scripts as much as possible.
  

3. Target 90% test coverage with automated testing. Even though test coverage does not tell anything about the quality of tests, it tells you where you need to focus more than others. Without test automation, it would be difficult to accommodate adequate testing in a sprint.

4. Inspect and adapt. Do not just stop at identifying the technical cause of a bug. Understand where (the quality gate) this bug could have been caught and improve the associated process.

5. Definition of "Done." Include all quality gates as part of your definition of "done."

6. Fix all know bugs first. Traditionally bugs are triaged based on severity as well as other works in the pipeline. The danger of doing this is that a seemingly minor/cosmetic bug could result into a large cumulative technical debt over time. Fixing bugs as soon as they are found can be less costly and time consuming than to wait for the right time to fix them.

7. Prioritize test cases. The intention behind this practice may not be apparent at first. The idea is that tests cost time and money. You can spend inordinate time in tests without getting any incremental/marginal value. Knowing how much testing is enough as well as what tests are more important than others not only will save time but also will ensure quality of testing. In order to understand what to test first, you could map your stories along two dimensions- frequency of use and risks of having bugs. The stories that gets used more often and also would have high risks should bugs are encountered needs to tested first and more.



I would be interested to know how you are doing QA on your Agile projects.

Stockholm Scrum Gathering 2008

I got back from Scrum Gathering in Stockholm a couple weeks ago. This was my first time in Sweden. It was colder than I expected. I went there over the weekend before the conference so that I can see some places around. Unfortunately, my luggage did not arrive with me. That kind of restricted my movement in that cold weather. Also, I realized that my session was the first one on the first day of the conference. I needed to prepare for my presentation...:-)

Overview of the Gathering. The conference was sold out. It was good to see a growing popularity of Scrum among the European companies. People came from many countries. I met people from UK (Conchango guys), Netherlands (Xebia), Denmark, Finland, Norway, and Germany (Siemens). The gathering was graced with the presence of Ken Schwaber and Jeff Sutherland. As always, the participants made the the conference come live through great interactions at the keynote speeches, and breakout sessions, as well as by self-organizing at the Open Space. The highlights of the conference from my perspective were the two great case studies presented by Xebia (Distributed Scrum) and Salesforce.com (Enterprise rollout of Scrum at Salesforce.com), the discussion on Agile Contracting at the Open Space, and Q/A session by Ken and Jeff to end the conference. Here are three pictures from the gathering,



1. Serge from Xebia on Agile contracting at an Open Space
2. Jeff and Serge on the role of product owner
3. People are mingling during a break

High Level Theme.It seems contracting is getting a lot of attention these days as more companies are adopting Scrum. The traditional contracting is based on waterfall model and it does not work for Agile projects. We expect to see some templates on Agile contracts to emerge over next year or so. Another overarching issue that everybody is grappling with is "Scrum" verses "Scrum Butt." Both Jeff and Ken talked about how to move from "Scrum Butt" to "Scrum." Jeff talked about the Nokia Test for teams to determine where they stand on Scrum maturity. Scrum adoption is a continuum and teams move along that continuum over time. All team starts at ScrumButt. Teams gradually move from "Scrum Butt" to "Pretty Good Scrum" to "Good Scrum" to finally "Great Scrum." In Jeff's word, hyperproductivity is achieved by "Great Scrum" teams. I am not sure about hyper productivity is sustainable in the long term though. Agile/Scrum development process is intense as it is. The goal is to find a sustainable pace that helps the team to maintain a sane lifestyle.

My Session on Agile QA. I will be honest that I was a bit nervous about how many would show up at my session. To my surprise, there was a great turnout at the session. A lot of good conversations happened during the presentation. I'll write a separate blog on the topic. Everything went well except that I was caught off guard when someone commented that we were doing "waterfall" when I was sharing how we were doing QA at a large government project as a case study on Agile QA. The "waterfall" sounds like a curse word specially at an Agile conference...:-) However, my key take away from my session was that I need to put more thoughts on how Agile QA would scale. In fact, scaling Scrum is not specifically an issue with just QA activities, it is an overarching challenge for all large projects. It is just that QA makes it more difficult. I plan to write about scaling Scrum in the future.

What's Next. Xebia case study showed how distributed Scrum and offshoring can produce great results. Agile/Scrum is increasingly becoming a critical element for making offshoring work. I look forward to seeing more case studies on the use of Agile/Scrum to make offshoring work at the next Scrum Gathering in Spring 2009. As for myself, I hope to have a session on Agile architecture and scaling Scrum.

Overall, I enjoyed the conference. The opportunity to visit beautiful Stockholm in itself was worth the trip. I would love to go back again. It is always difficult to visit a place alone. However, I enjoyed my walk-around the old city. I wish I had more time to visit the islands. Next time...:-) Here are four more pictures of Stockholm,




1. Bustling evening in the alley of old city of Stockholm
2. The Royal Guards at the Swedish Palace
3. The sun setting in the alley



Finally, this picture is for people in Bangladesh, do you recognize the logo in the picture?

Look for "P.H.P." Skills in Your New Hire for Your Startup

Every entrepreneur agrees that getting the "right people" on board is critical to the success of any Startup. At the same time, finding the "right people" is one of the most challenging job that any entrepreneur does on a regular basis. Lately, I have been thinking a lot about what I look for in a prospective hire for my Startup. Probably because I am hiring for two key positions- one at my offshore and one at my on-shore office at the moment.

For a long time I could not quite put my finger on the qualities I implicitly look for in the new hire until recently when I had my "aha" moment. I wanted to capture my "aha" moment by sharing with everyone in the same boat. It is a simple list of three qualities in order of priority- Passion, Humility, and Patience (PHP). I'll explain why and how these three qualities help identify the right people for your Startup. But before I do so, I want to list a set of complementary reads from OnStartups blog that I would highly recommend to anyone in the same boat . These are,

• Startup Hiring: An Entrepreneur Disagrees With Entrepreneur Magazine

• Startup Hiring: 6 Subtle Signs You Might Have A Winner

• 5 Quick Pointers On Startup Hiring

• Startup Hiring: Why You Should Date Before Getting Married

• Startup Hiring #2: Dealing With The Dating Period

My thoughts on hiring complement the thoughts you would find in the above writings. Now, let me explain the PHP qualities:

Passion. Startups are born from founders' passion for an idea. The people who join them in that journey must need to share the same passion, if not at the same level. It cannot be "Just another Job" (JaJ). A passionate person is always resourceful to acquire and apply skills to help morph and implement the idea.

Humility. A group of passionate people need to work together to achieve the common goal. Without the teamwork, passion alone cannot get us to the destination. People need to put the "team" ahead of the "self." They need to find a way to feed their individual ego (we all have one...:-)) from the team effort and resulting success. If "Passion" is the glue to bring everybody together around a common vision and mission, "Humility" is the chemical property that gives a person the ability to become an integral part of that team. The higher the degree of humility, the greater the chance that the new hire will make the team better.


Patience. The path to success is anything but smooth. Many people may have the passion and humility, but may not have the stomach or patience to withstand bumps along the way. To continue on my analogy of glue, "Patience" is the other chemical property of the glue that allows a person to withstand the ups and downs that awaits him by being at the Startup. Working at a startup is not for the faint of heart. The higher the patience, the higher the chance that the new hire will survive the inevitable wild ride of a Startup.

Now the question is how do we determine that the prospective hire has these qualities in him or her? Great question. I don't think there is any silver bullet. The key to asking questions in the interview is to probe on what the person did, not what they would do in a given situation. The trick is to guide the conversation in way so that you can capture signs/clues that help you understand the degree of "PHP" qualities your prospective hire posses. It may seem very easy. In reality, it is difficult, trust me. Unfortunately, most of us end up asking leading questions and the interviewee ends up answering what the interviewer wants to hear.

At the end of the day, you need to develop your own style that works for you. You can also read "Startup Hiring; 6 subtle Signs You Might Have Winner" to learn about how Dharmesh Shah does the same. I am interested to learn about your thoughts on the topic and what three qualities you think are important in your hire for your Startup.

New Frontier of Agile Contracting

I blogged about how I always shy away from fixed-price contract. As a result, I pass up opportunities where clients are not comfortable working on a T&M basis. So, I have been looking for ways to accommodate clients concerned (apprehension about not knowing the total costs) about T&M and find a happy median. I was happy to learn at the Agile 2008 conference about a few approaches that some companies are trying. I thought others might be interested to learn about these novel approaches to contracting.

Money for Nothing and Change for Free. This is the most intriguing of all the variations I heard about. Jeff Sutherland, co-creator of Scrum, shared this interesting case study of a company that was contracted by a client to do a large project. The RFP was awarded based on waterfall approach. The company estimated the project to take 18 months and would cost $1.8MM. Essentially, the project was awarded as a fixed-bid contract. However, they negotiated with the client that they will do the project using Scrum. And they added an innovative clause to the contract that said if they delivered 80% of the intended value from the project, the client would have the option to end the contract and would pay the company (the vendor) 20% of the remaining contract value. Guess what happened? The company was able to deliver 80% of the value in 3 months into the project and the client was happy to pay 20% of the rest of the contract value. Both parties were delighted. Pretty neat, hah! So, to generalize this approach, we could bid for projects in the traditional way (i.e., fixed-bid), but add the following clauses,

• Client can change requirements for free ("change for free"). Traditionally any change to requirements results into repricing that usually causes the project costs to go up.


• Early delivery results into a bonus for the vendor which is equal to 20% of the remaining contract value. This is what is called "money for fee."

Hybrid Contract- T&M Definition Phase, Fixed-price Delivery Phase. Essentially vendor will work with the client to on a time and material basis to build the product backlog (requirements) and produce a high level estimates and validate assumptions. Once both parties are comfortable with the definition of the project, the project enters into a fixed-price delivery phase. This will work when the product backlog can be defined reasonably well in a reasonable time-frame as well as the the development team is experienced in the domain and technology and that the development team knows its velocity.

Hybrid Contract- T&M Delivery of Phase 1, Fixed-price Delivery of Subsequent Phases. With this approach, the project is broken into multiple phases. The first phase is done on T&M basis with CAP maybe. In this phase, both the client and the vendor collaborate to identify and address domain as well as technical unknowns. By the end of the T&M phase, the project is expected to have set on a predictable course so that the vendor can price out the rest of the project with reasonable confidence.

Fixed-rate per Unit of Work Contract. This approach requires the vendor to have a stable team that has a well established velocity and also familiar with the technology and domain. The contract is awarded based on an expected rate of work produced per iteration at a fixed cost. The unit of work can be in story points or function points. This would be the nirvana of agile contracting. This could transform software development service into a "Service on Demand."

There are variations within each of these contract framework. I would be interested to hear from others on this topic. Drop me a few lines.

Agile 2008 Conference

I attended the Agile 2008 conference in August. This year it was in Toronto. I like the city. The weather was nice and the conference was lively. This was my first time at this conference. Apparently it is growing every year, which is indicative of increasing popularity of Agile philosophy of software development.

There were many interesting sessions at the conference. The most notable of all was the keynote speech by Robert Martin, a.k.a. "Uncle Bob." I was already a fan of his writing. However, I did not have the opportunity to see him in person until this conference. If you haven't heard him speak, you should if you get a chance. It is not only the content of his speech that is interesting and insightful, but also his charismatic delivery is a thing of entertainment. This time he talked about the importance of "clean code," a topic of his new book. I am planning on getting it soon. In his speach, he pointed out how XP has become all about just 12 engineering practices (e.g., "thou shall pair program, thou shall have common code ownership, ...") and Scrum about certifications. The gist of his speech was neither approach is self-sufficient. He urged everybody to focus on writing clean code and not to take short-cuts. In his words, "Craftsmanship over Crap." Here is a picture of him speaking at the conference.

Equally interesting (but not as much entertaining) was another keynote speaker James Surowiecki, the author of The Wisdom of Crowds. He talked about how "wisdom of crowds" may apply to software development teams and more specifically to self-organizing teams. The gist of the talk was that a team carefully put together can collectively make better decisions than any individual experts. He elaborated this with a few interesting examples form real-life experiments. It jives with my thoughts on how a team can collectively act as an Architect.

However, the closing keynote speaker, Alan Cooper, got less than enthusiastic reception for his speech. He shared his unique view of software lifecycle, which is a hybrid of agile and waterfall. He also highlighted the importance of a less understood role "interaction designer" in the context of this software lifecycle. Even though I liked his thoughts on the role of an "interaction designer," but I, like most of the audience, was skeptical about his idea about the software lifecycle. You can find his presentation here. I'll let you be the judge for it.

I am already looking forward for the next year's conference already scheduled to be held in August in Chicago. I hope to see you there.

Leadership on a Self-organizing Team

Self-organization is at the core of a successful Agile/Scrum team. When a team is transitioning from a traditional team structure (hierarchical) to self-organizing team (flat), I have seen the team not sure how to accommodate traditional roles like team lead or architect.

Let's delve into how a traditional team works. The single point of accountability lies with a team lead who is expected to ensure the success for the team. He/she decides who does what and delegates work accordingly. Even though, a team lead may involve team in making these decisions, more often than not it is a one way street. When we are accountable for the outcome, instinctively we feel compelled to force our way of doing things. And the rest of the team takes a back-seat in the whole process.

In a self-organizing team, on the other hand, each team member is accountable for his or her work and collectively accountable for the success of the team. This collective accountability takes the center stage in a self-organizing team. There really is no single person to point finger to. As a result, the team collaboratively decides who gets to do what. It may be driven by the current skills, experience, as well as the need for cross-training and self-interest to learn new things. The leadership is not bestowed by a title, rather is earned over time through performance. In fact, the leadership becomes distributed based on specific expertise. For example, a team member might show a lot of interests in database technology. He/she establishes himself as the SME (subject matter expert) by voluntarily solving database related complex issues facing the team. Similarly, another person may show greater understanding of OO design and programming and the team might start to defer OO matters to him. By effectively bringing these individual expertise together, a team as a whole, could work as an architect and come up with a good architecture. This may not happen with a team that is working together for the first time. The Scrum Master will need to help the team become self-sufficient and self-confident to fill in the traditional roles of an architect or a team lead.

So, am I saying the need for an architect or a team lead is gone? The answer is "Yes." A team may choose to make the SMEs the spokespersons for the team for topics. The topics could be technology, or business. The SMEs will handle all questions/queries related to their own topic. It needs to be published to all stakeholders. Each SME will need to keep the responsible enterprise architect in the loop for his or her area of interest. For example, if the issue at hand is security related, the team SME for security will work with the enterprise architect.

I know you might be thinking that this is not possible on a real project. I agree it may not be the case when a project team is working together for the first time. We may need help from a solution architect who could help the team to weave all discplines into a cohesive solution. However, the goal of the solution architect should be to help the team get to a point where they can collectively drive the solution forward. The first step towards achiving this "team nirvana" is to put together a team with a collective knowledge and experinece covering all aspects of the project.

User Stories vs. Use Cases

I am consulting on a large government project, where we are using Agile/Scrum. Lately we have been having passionate discussions around whether or not User Stories are enough for capturing requirements for large projects. My answer is yes User Stories are good enough for capturing requirements for any size projects. However, the product owners on the project are of the opinion that User Stories need to be complemented with Use Cases. We couldn't come to an agreement and the discussions continue. I recently came across Jeff Anderson's blog that exactly captures the questions/concerns that my co-workers on the project have raised. So, I thought let me share my view that I already did with my co-workers.

User Stories do not provide context. The way to add context to user stories and linkage among them is by associating stories with themes/tags. We could either associate a story with a single theme or multiple themes. For example, say a story is - "I as an Admin can approve registration requests." We could associate this story with "admin," and "registration." Additionally, we could associate the story with "architecture." Maybe this is the first story to address how we would implement admin module as well as security roles. So, it is architecturally significant. Later we could filter all stories that are architecturally significant. The themes can also be organized in an hierarchical fashion. However, I would argue that the same thing can be accomplished without explicitly creating this hierarchy. This will make life much simpler and provide greater flexibility in organizing and viewing requirements.

User Stories do not capture appropriate level of abstraction. I agree that the implementable stories might not be suitable for tracking and reporting to sponsors and management. However, if we think about how stories naturally evolve, then we would realize that most stories start at an epic level. Then over time, we break the epic stories down until we find implementable stories. The epic stories should provide the abstraction that we are looking for. We could report to the stakeholders/management on progress at the level that we feel appropriate. Of course, we would need a good tool support to help us with tracking and managing the relationships among stories. Watch for ScrumPad to provide this capabilities soon. However, we could also do this using plain old Excel.

User Stories do not capture enough details. Well, we could capture any and all information related to a story that we could in a Use Case. The amount and level of details that we should capture depends on many factors. The trick is though the way to put the details and being able to understand when is too much or too little. This is more art than science and depends on individual experiences and preferences. However, the intention should be to make a story card a starting point for conversation and collaboration between developers and product owner. I see a tendency among product owners coming from Use Case background to put in a lot of details as an alternative to conversation. The fallacy in that is the details are known, understood and can be unambiguously communicated in written format. We all know how that works.

However, on our project we are mapping stories to Use Cases where we need to interact with an external team that is not familiar with User Stories. In general, User Stories can be used to manage requirements for any size project.

Architecture on Agile projects- define a baseline architecture

One of the questions that frequently comes up when people move from waterfall to iterative incremental software development (Scrum, XP), is how architecture fits into this new world. Most people of the opinion that architecture needs to be in place before you can start iterative incremental development. My answer is yes and no. I will try and explain what I mean by that. This will be first in the series.

Whether we are building a simple application or a complex enterprise application, we need to have a starting point. The starting point of an architecture is defining a baseline.

The baseline architecture. Yes, this needs to be in place before we start our first sprint. In sprint 0, we will define our baseline architecture as we build our product backlog. Some of the things that are included in this baseline at this stage are,

1. What software platform to use? We need to decide whether to use Java/J2EE, or .Net, or Ruby on Rails, or some other technology. We also need to select what version of the technology to use. Do we want to use the latest stable release or beta release of the technology? The decision depends on many factors. Among them are type of application, existing standards (companies may already have made this decision as part of enterprise architecture), experience and expertise of the developers as well as support people, strategic partnership etc.


2. What hardware platform to use? This decision depends on the decision from the previous question. If we select .Net, then we all know we will have to use Wintel hardware. If we select Java or RoR, the choices are more- Lintel, Sun, AIX, etc. However, we most likely will choose Lintel. We may also have to make a decision about the vendor we will get the hardware from. Most of the time companies have existing relationship with a vendor.


3. Where to host? We know what technology we are using as well as which platform to run the application on. How about where will the servers be located? Do we want to put the server in-house or do we want to outsource them to one of those hosting providers. If we already have an in-house data center, the answer is clear (even though I am not sure why most companies would want to continue to own and operate their data center now-a-days). If we decide to go with a data center provider, we have a number of options at our disposal- shared, virtual dedicated, dedicated, and cloud. However, who do we select as our data center provider depends on whether or not the hosting provider can support the growth of your application (scalability) in addition to pricing, and reputation. We do not need to know the exact load profile of our application to understand how the platform will scale. But, we need to have an idea about the expected growth of usage of the application and the type of users.


4. What architecture style and pattern to use? Architectural style sets the tone of the solution whereas Architectural patterns, like design patterns, provides a proven skeleton for implementing the solution following the style. The most popular style is n-tier architecture. In recent years, we have seen growing popularity of SOA (service oriented architecture), and RESTful(representational state transfer) style of architecture. However, the mother of all architecture pattern is MVC- model, view, controller. We cannot go wrong with this and it is applicable to most situations. Other patterns to consider in conjunction with MVC are hub-and-spoke, pipe-and-filter, and publish and subscribe.


5. What technology tools to use? Most likely you would build the solution from using one or more off-the-shelf products/components/tools, commercial or open-source, instead of doing everything from scratch. Some of the type of tools that you would want to consider are- MVC framework, ORM framework, logging library, instrumentation framework, rule engine, workflow engine, integration engine, BPM, reporting engine, ETL etc. Some of these tool selection may involve a formal vendor selection process. However, the baseline architecture should include a short list of 2/3 choices.


6. How to handle security of the solution? I have listed this as a separate item to consider for the baseline architecture because of its importance. Even though we all understand the importance of security of any solution, we tend to put it in the back burner. We developers usually like to use adminstrative access (turn off security) to avoid the headache of dealing with security issues when we write codes, which is at odds with "Principle of Least Privilege." The best practice is to start with no previliage and open up just enough to get the job done. The baseline, at a minimum, should define how to perform authentication and authorization as well as data encryption. For example, we need to decide whether to use windows or database or LDAP for authentication and authorization. Also, we need to decide whether to encrypt communication and when to encrypt (e.g. HTTPS).


7. What development tools to use? Even though it is not part of the solution and hence is not part of the architecture, it assists in implementing the architecture. Also right development tools ensure efficient and quality delivery. The is why typically the architect is on the hook to drive these decisions. The tools are SCM (software configuration management), IDE, testing (unit, integration, functional) frameworks, build tools, project management & collaboration tool (e.g., ScrumPad), and design tool (e.g., Rational Rose, Visio).


8. Setup of separate development, and test environments. Typically developers write codes on PC/laptop and then push the codes to integrate and run in a separate environment for testing before the application can be put in the production environment for actual use. The test environment must replicate the setup of the production environment. The only difference between the two environments is that the production is setup to handle the expected load. That is why it is important to define a unit footprint of the solution to help setup the test environment. This also allows you to define how to scale your solution in production as well as determine when to scale (in terms of the load supported by the unit of the solution). For example, say your unit of the solution can handle x load, and you need to support 5x load, you know you need to get 5 units of your solution. Even though it may seem common sense , you will be surprised to find out that how many companies do not have this in place. Yes, there are situations when you cannot have the test environment exactly replicate the production environment. This is usually the case when legacy systems are involved. For development environment, we also need to make sure that the developers can integrate with rest of the codes easily. We need to make sure a developer can have access to a representative instance of each element of the architecture. For example, if the solution is using SQL server enterprise edition, the developer may choose to use SQl server express. Sometimes, we may not have separate instance for each component of the solution for individual developers, we may need to make an effort to come up with a work-around. For example, you may need to integrate with a Mainframe application, but it may not be accessible from a developer's laptop because of the absence of network access. You could use the test environment to allow developers to access the mainframe application from their laptops.

I know some of these are common sense elements of any solution, yet I am surprised to see how many projects do not have these in place. We should spend a lot of time to define the baseline. Some of the decisions are related and one decisions will help narrow our choices for the others. Others are purely individual (or organizational) preferences and we should be fine either way. Once we have the baseline defined, the detailed as well as additional elements can be iteratively fleshed out. I will discuss how to evolve a baseline architecture iteratively.

Freelancers vs a Team?

We have worked with a few clients who came to us after having bad experiences with their freelance developers. The common thread of all these bad experiences is one thing- lack of proffesionalism on the part of the freelancers. Don't get me wrong, some of these freelancers are technically good. In all cases, the motivation to hire freelance developers are usally is to save money. However, the savings become illusive when the projects start to go south. By the time we come into the picture, it becomes tricky for us to help. Sometimes we find the work done so far by the freelancer is missing things that the client thought already done. Sometimes quality of work such that it may make sense to start from scratch. It is financially as well as physcolgically difficult for the client to receive this meesage.


Why freelancer does not work for software development. No offense to all freelancers out there, software development projects are not suitable for freelancing. Software development is a team work, and must be done by two or more people. We developers are bad when it comes to following best practices on our won. We tend to fall prey to our own vices. But when we work in a team environment, that forces us to follow a structure and best practices, whether we like it or not. Besides, when there is an extra pair of eyes, design and quality of implementation tend to be better just by having different perspectives. It also provides the needed check and balance.


Checklist to help determine freelancer vs company. I have made a checklist for those who are thinking of outsourcing software development projects to help determine the right approach.

• My project is critical to my business success.


• I have a budget > $5000.


• I know what I want at a high level, but not sure about the detailed features of the software.


• I do not have experience managing software project.


• I do not have a high level understanding of technology involved.


• I do not have experience with the business domain.

If one or more of these are applicable to you, you should consider hiring a company. The reasons should be self-explanatory. Next time I'll share another checklist to help with selecting a software development company.

Economies of experience and Agile development

We always hear about economies of scale and scope. Do we ever think about how do these two concepts apply to software development? Let's consider economies of scale first. If we get more developers, we maybe able to negotiate lower rate from our vendor. Another way to look at it is the larger the project size, the lower the rate per hour of development. What about economies of scope? When we develop software, we could design it in a way so that we can end up with a set of reusable components. We could then use these reusable components and build our next software for an adjacent/related domain quicker. Building the first application will cost more. The successive applications should cost less. Thereby we achieve economies of scope. Here is one more reason to ask your boss to invest in OO on your project.

OK, you probably already know this. The reason I brought this topic up is to introduce another related concept that I call economies of experience. I don't hear anyone talk about this in the software industry. In manufacturing, there is a similar thing called experience curve. How exactly does economies of experience apply to software development? When we hire developers for a project, we all look for individual experiences in a developer with the technologies and in the domain the project is. The longer a developer works with a technology and in a domain, the more productive he/she is with that technology and in that domain. We, however, never think about experience at a team level. Developers maybe individually very productive, but when they work as a team it is completely different ball game. Like an individual, a team also needs nurturing to be able to unlock its fullest potential. In the abscence of active coaching and guadanice, a team can end-up crashing and burning.

Every team has a unique DNA. The DNA emerges over time. The longer a team works together, the more productivet the team gets. That is why Agile/Scrum uses a drag factor (negative productivity) in estimation. We increase the total estimate by this drag factor. A team working together for the first time may start at 40% drag or 60% productivity. Usually, it takes a year for a team to reduce drag to 0 or reach 100% productivity. Anytime a team member leaves or a new team member joins, productivity of the team goes back down.

In traditional project management, resource planning is done using staggered or just-in-time approach. We bring a developer on board only when we have a need for his/her skills. The thought there is that we would save money by engaging people only when they are needed. However, this (the savings) is seldom achieved. The further along the project, the higher the impact on team productivity for adding a new person to the project. That is why Agile projects make a reasonable effort to recruit the full team upfront with people who will be able to wear multiple hats. In the long run, the savings come from higher overall team productivity and utilization.

Conundrum of fixed price contracts for software development service?

I get this from our prospective clients a lot-

I need to know how much it is going to cost me to build my software.

The conundrum. I understand where they are coming from. After all budget is always finite. We do not feel comfortable committing to buying something without knowing the total costs upfront. While it is possible for fixing the price of many goods or services, it is very difficult, if not impossible, to do so with software development service. The simple reason is that the estimates are as accurate as the requirements. It is not possible to define everything one wants in his software with all the details. That is why buyers try to put many clauses in the contract to move the risk over to the service providers. On the other hand, the service providers try to push the risk back to the buyers by requiring the spec to be detailed and complete. We all know words are imprecise. Hence both parties end up with a contract that helps none. So what is the remedy?

Agile philosophy is in rescue. The concept is simple. The cost driver of software service is time. Since the budget is fixed, the total development time is automatically fixed for a specific rate. The only thing that we can play with is the scope of the work. Yes, I know who wouldn't like to have everything, but the reality is we need to restrain our wants. How do we do it?

Enter the world of no long-term commitment. The client creates and manages a prioritized list of things/features that he wants in the software. In Scrum, we call it product backlog. The developer works his way from the top of this product backlog, and delivers working software every 1-4 weeks (iterative incremental software development). That way the client is sure that he is getting the most value for their bucks. Yes, we can still give our clients a high level costs estimate in terms of number of sprints it might take to complete the project with an understanding that it would change as the project progresses. As the developer makes no commitment to any effort estimate, so does the client make no commitment to any budget amount. Both work collaboratively to manage the scope (the product backlog) to meet the goal and the budget. Every two weeks (assuming the development cycle is 2 weeks), the client can change the course of the project based on the current state of the working software in progress as well as the business conditions.

What should you do? I know this could be unsettling for many who are used to the old ways of managing vendors and projects. Believe me, if you try it out on your next project, you would not want to do it any other way. It is truly liberating when the client and the service provider work collaboratively to achieve a common goal without worrying about legal documents.