I believe in aeroplanes

Thu 27 January 2011

How can you be sure that something is true?

Actually it is surprisingly difficult... Do I mean true as in fact or true as in 'I believe this'? Maybe we're in the matrix, and everything is a simulation... maybe the people you trust are lying.. or maybe they're just wrong...

Who cares? I think we all should.

Belief Systems vs Science... A lot of what we think is true is based only on Belief Systems (B.S.), not facts. It's surprising how far B.S. can take you ... communism, capitalism, religions, alternative medicines ...and most 'methodologies' all rely on B.S.

The alternative to B.S. is science. But strangely enough science doesn't actually claim to prove facts. Scientific laws and theories are explanations for how and why things happen, supported by the observed data - usually from many independent experiments - but they may be later modified or disproved by future research. Non-scientists often misunderstand theory in this context, e.g. they argue 'it's only a theory, so alternative theories are equally valid'

Science relies on the principle of testability and disprovability. Conversely, most B.S. ideas have not been properly tested - many are not actually testable, and are therefore not disprovable.

A worked example... Quantum mechanics is a scientific theory. It's completely counter-intuitive... for non-scientists it seems utterly nonsensical, check out Schrodinger's cat for example.

However, quantum is supported by many experiments. And all electronic devices exploit quantum theory.

So if you believe in aeroplanes, you're by implication relying on quantum theory. And so are all of the deeply religious ideologists (no matter how traditional) who drive cars, cook in microwave ovens, surf the web and make phone calls.

So while B.S. can be extremely useful for influencing people, the usefulness of B.S. for transportation, cooking and long-distance communications is zero.

Back to the point... Enough philosophy - how does this relate to software projects? Well, most management and software methodologies are based on B.S., as opposed to science.

The currently fashionable Agile crop is no exception. This may or may not surprise you, given

  • the 'thought leaders' appear highly qualified
  • the Agile literature (books, articles etc) use references and citations just like scientific texts
  • they talk about 'metrics', 'studies' and 'research', again just like science

Let's take these one at a time...

Expert Consultant with MBA/PhD/30 years experience... Qualifications and experience can be falsified, exaggerated or irrelevant. For example, I know of a chap who wrote a best-selling diet/lifestyle book, and claims a Ph.D in nutrition. Obviously the qualification helped the book sales... but the trouble is there's no record that he ever published an actual Ph.D thesis... did he buy the qualification from some dodgy internet site, or just make it up?

It works for Yahoo/Google/Toyota/Me... Anecdote is not data. If I tell you my grandma beat cancer by bathing in goats milk, will you try it? The management practices at W.L.Gore are often held as proof that self-managed teams work better than old-fashioned command and control management (do I need to provide citations, or will you believe me anyway?). This might even be true in some contexts - but it's clearly not true in general. No sane country would consider a self-managed soldiers, surely? What about self-managed teams running a nuclear reactor... is that ok by you?

The Survey says... Surveys are not equivalent or comparable to scientific research. This should be obvious - folks will come up with all kinds of nonsense in response to survey questions, for all kinds of reasons. If your job title is Widget Specialist, are you likely to say that Widgets are a thing of the past? Equally if your survey doesn't support the result you are hoping for, will you still publish it?

Metrics... Defect rates, velocity, function points, KDSL (thousands of deliverable source lines) all sound like good reliable numbers, don't they? So 'Agile projects have lower defect rates' sounds like it must be based on facts, right?

Sadly, probably not - for lots of reasons, including but not limited to:

  • Gaming the numbers: if folks are assessed/paid based on a metric, they will find all kinds of 'interesting' ways to improve the metric. For example, splitting code over multiple lines to increase KDSL, splitting/adding stories to increase velocity.
  • Measurement fallacy: just because you're measuring it, doesn't mean you know what's going on (a great demo of this is Deming's Red Bead Experiment)
  • Subjectivity: my idea of a user story may be several stories for someone else's project. Similarly what I call a defect, you may call new/changed functionality. I might even change my own definition of a defect during a project - particularly if I'm assessed/paid based on defects :-)
  • Other factors: take project context, for example - defect rates on my one-man ruby/rails/web project are clearly not comparable with a multi-site team developing an embedded system.

To sum up... Just because lots of folks have bought into something, doesn't make it right. As the old joke says... "Eat BS... 50 billion flies can't be wrong"