Lies, Damned Lies, and Software Development Estimates

Yes, that might be an odd headline for me to make, given that I actually run a software development company.

Or is it.

I was catching up on my little corner of the blogosphere and finally arrived at Reg’s most recent post on theory D, theory P and his subsequent follow up.

Theory D people are those who believe that software development is a Deterministic field, and Theory P people are those who believe that software development is a Probabilistic field. Theory D people believe that you can calculate deterministically the effort required for any given software development project. Theory P people believe that it’s a more organic process where needs and desires drive the outcomes. Take a look at Reg’s posts for some background on this — it’s OK, I’ll wait…

…ok, so here’s my thing. I believe everyone starts as a Theory D person, regardless of the field. To an onlooker, it’s hard to see the complexity of a field especially when observing an expert. And as humans, I believe we have a tendency to underestimate the complex as a survival mechanism. If we really understood how hard it is to do something, we may never try and we would not advance.

Consider the idea that the more you learn, the less you know. The highest formal educational designation you can attain is a PhD, which is a Doctor of Philosophy. Your understanding of your field moves from the Deterministic to the Probabilistic, the physical as well as the metaphysical, where you must argue your theories with the rigor of the scientific process and human creativity rather than a calculator.

The more you learn about software development, the more you migrate towards being a Theory P person.

I believe this is one of the key challenges of being in the Software Development Professional Services business. You are perpetually teaching people along the road from Theory D to Theory P.

So back to my shamelessly sensationalist headline. Why do we continue to do software development estimates?

It all comes down to the fact that software, in the end, must perform a very practical and ultimately mundane function. A function that may be perceived as simple to the stakeholders. And that function must be in place in order for the rest of their business to function. It is an imperative.

As a business imperative with an associated cost, the cost must be justified. The business must make the decision whether it will profit from or lose to the imperative which is a direct relation to its cost. A low-cost project will be easier to justify than one with a large price tag.

While the professional services salesperson tries to illustrate value to the prospect, the prospect is focused on the imperative which appears simple, and the two negotiate whether the project is feasible and at what cost.

This is where I apply my favourite question, “What is the smallest simplest possible thing we can build that will still meet the majority of your needs?“.

The reason this is my favourite question is two-fold:

1. we do iterative development, and we’re looking to start small, deliver quickly, and do future projects to build on it
2. small projects are easier to estimate because they have fewer factors affecting them, and you can approach a level of understanding that’s aligned more closely to what’s going on in Theory D people’s heads

I will rarely estimate beyond the first 1-2 iterations. There is simply too much uncertainty beyond this point. Sometimes I will steer discussion towards a larger target budget, but there are no promises other than my team will work hard in the best interest of the customer, and deliver useful software at each iteration.

If pushed further, I move to a risk based model. If I’m going to take on the risk of someone else’s project, they’re going to get an estimate that reflects that. The bigger my risk, the more expensive the project. This is no different than the insurance industry. Now you wanna talk non-deterministic… :)