Friday, February 05, 2010

Retrospective on ScrumPad Product/Market Fit

As I am beginning my journey of Lean Startup with our ScrumPad service, I plan to share my experience along the way.

The first stop in the journey is "Product/market fit." For the uninitiated (I am not an expert either, learning by doing), all it means is to prove that there is a large enough market for the product or service that I am building. The goal at this stage of a startup lifecycle is to identify/define that market with specific details- who are the users, why they like our product, and the business model that would work the best, and a repeatable sales model.

The best way to adopt the Lean Startup model is to first look back to understand where we are (since it's been 18 months that we are working on ScrumPad). During this time, we did not apply the Lean Startup model (were not aware of it). We only focused on product development using Agile software development model (the only good thing we did). Customer development was largely absent. I haven broken the retrospective in two parts- 2008 retrospective, and 2009 retrospective. And I have structured it in light of Lean Startup for easier understanding.

Version 1- Free ScrumPad: 2008 Retrospective
What was our hypothesis?
All started with a itch of ours. We looked for a tool that fits our needs. We found two- Basecamp and Version One. But, none was what we were looking for. After trying both, we had decided to build one that we liked. So, since we liked what we had built, we thought there must be more like us who would also like ScrumPad.

How long did we spend to validate this hypothesis?
6 months.

What did we do during this period?
  • We primarily focused on adding functionality (product development, not customer.). We thought more functionality = more satisfaction = more users. 
  • We were releasing incremental version of ScrumPad every two weeks using Agile/Scrum process.
  • We did not do anything on the customer development front. Did not do any marketing. Did not talk to customers for feedback (did not get out of the building). 
  • We completely deferred design thinking that design and usability is secondary to functionality. We thought once we have the functionalities in, we could always add design and improve usability.
  • We did not charge for our service. We thought we need to first build out the full product (not thought about an MVP) before we can start charging.
  • We did put some basic tracking- Google analytics as well as some custom ones (i.e., number of projects, users, and overall project activity index that we defined). It helped us detect some early issues with signup and getting started with ScrumPad. 
What was the result?
We did get a constant stream of SEO traffic from search engines. This traffic resulted in a trickle conversion 1 to 2 every week. Received some unsolicited feedback, primarily from those who liked it. However, when we started charging in early 2009, most of the users from 2008 stopped using the service.

  New   Bounce Avg. Time   Pages 
Visits     rate     on Site       per visit  Conversions
3,521     48%   00:02:50            2.61          5%

What did we learn?
Our product is better than the opensource counterparts. However, as a paid service, we did not learn anything about whether we have a market yet as a paid service.

Version 2- Commercial ScrumPad: 2009 Retrospective 
What was our hypothesis?
If we shift focus from functionality to user-interface and usability, there is a market for our service for a $21/user/month with single pricing model (no tiered pricing). I did a pricing analysis of our competitors (I must admit my class on pricing for my MBA from UNC helped).

How long did we spend to validate this hypothesis? 
All of 2009.

What did we do during this period?
  • We changed our logo, updated the UI, and continued to tweak the interface to improve usability. 
  • However, we spent a significant amount of time in refactoring (rework) to fix the sprawling code-base that resulted from the quick piling up of functionality. 
  • We started actively asking for feedback. We in fact incorporated a short exit interview during the cancellation process. 
  • We started charging for our service. 
  • We still did not do any CPC marketing. We wanted to focus on word of mouth. We added affiliate program instead. 
What was the result?
We continued to have a constant stream of SEO traffic from search engines. Even with pricing in place, we saw significant improvements in all metrics except trial conversion rate. We saw a drop in trial conversion rate (by 1%. I think it makes sense since its no longer free). With "trial to subscription" business model, we finished the year with a few paying customers, but no new customers through our affiliate program.
 New     Bounce    Time      Pages per        Conversion
Visits  rate Avg.   on Site      visit         Trial   Trial to Paid
   7,578    38%    00:04:23      3.72           4%          4%

What did we learn?
Since we have customers who are now paying for our service, there exists a market who are willing to pay for our service at the price-point we are offering now. However, we do not know who they are and how to reach them, and why they like our product over our competitors. Also, the metrics suggest that we have room to optimize the conversion rates through A/B testing of landing page and other aspects of the site and the service. Affiliate program was too early to add.

In hindsight, I believe what we achieved in 18 months or so, we could have learned even more in 6 months or less using Lean Startup techniques. More on what's next from here. 


  1. Thank you for sharing your experiences. I wish you success as you get out of the building.

  2. Kent, thanks for stopping by and for your well wish. I am a fan of your work for along time. It's an honor to have you here.

  3. Syed, nice post.

    Especially this line stood out: "Did not talk to customers for feedback"

    I went through this exact pattern in my previous company (also a startup). I think this is a big problem in the normal Scrum process that happens to a lot of people.

    I blogged about it here - and here -

  4. Hi Syed,

    Thanks for sharing. Very informative for start-ups and entrepreneurs.

    Best of luck!


  5. Siddharta, good to know that we share the same experiences. Checked out your slides. I look forward to the video...:-)

  6. Old maxim - clients don't buy features; they buy benefits. Similar old maxim - clients don't buy products; they buy the anticipation of some feeling.

    Can you test a "feeling" hypothesis?

  7. Jim,that's what we plan to do. We just incorporated Sean Ellis's customer development survey. It tries to measure users' feelings. Hopefully most of our users will take the survey...:-)