Technical Debt and the Cost/Benefit of Knowledge Retention

A rather rigorous, Financial-sounding title for a high-concept line of thought …

Thanks to Jeff Atwood at Coding Horror, for calling my attention to this article by Martin Fowler on Technical Debt:

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

Now, before you write off Cunningham as a techie snob or an academic hold-out for unattainable perfection, listen to this healthy dose of reality …

The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity, developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.

I think most of us have seen this phenomenon before; sometimes it manifests as an open willingness to trade quality as just another feature (as measured by the amount of testing before code is put into production). Documentation is another common sacrifice – too often we accept e-mail summaries or PowerPoint outlines as a reasonable facsimile for knowledge capture.

You’ve probably seen this phenomenon where you work, and not just in your IT organization. Many areas of the business will rationalize over-budgeted schedules by summarizing critical findings in a brief email – or, worse, in a Status Update Meeting. “This is an expensive meeting”, I might quip upon entering the room, seeing the conference table ringed with upper-and middle-managers, each weighing in with their understandings and opinions. Don’t misunderstand me – these are typically very effective conversations, with exactly the right people; the folks that know and live the issues, and fully understand the implications of any process change. But my witty entrée was tragically accurate; the understanding and decisions developed at this meeting are too often lost a few minutes after the meeting ends, ideas with a half-life approximately 10 minutes into the start of the next meeting.

Think of it as a knowledge expense (vs. depreciation, as value is lost rather quickly). The expedience and effectiveness of face-to-face communication, with everyone in the same room hearing the same thing consistently and able to ask questions to validate their understanding, typically does not scale beyond the attendees. It’s like listening to a band vs. buying the album (ah, more poetic than downloading …).

In his article, Atwood continues along the Fowler / Cunningham thought process, discussing the need to budget a certain amount of time to pay down our technical debt by going back and finishing that unfinished work; document the things that you sloughed over, rework the inelegant parts of your database schema re code interfaces that rely us a little bit too much on assumptions.

The same can be said for process design and problem solving sessions – remain aware of your level of knowledge debt and budget time to document your findings. I like to call these chunks of captured knowledge “white papers” – I’ll pause while you admire that stunning originality, but there’s a method to my blandness. Calling these things “white papers” helps folks understand the purpose and value of such a document;  reasonably short and idea complete. The sweet spot seems to be two to four pages, well-organized, not too wordy, but clear enough that it remains effective months after the design or process rework sessions took place.

Just remember, organizations do the expedient thing all the time, streamlining meetings and decision-making by going light on the documentation.  Every once in while, you’ll pay the cost of rework and rediscovery; as our experience grows, and our patience for such “wasted effort” grows thin, task effort times will increase as we invest a little bit more time in better, clearer documentation.