There’s no shortage of assertions that conducting postmortem analysis of any decently-sized software development project is a good thing (there are academic papers, even). However, what often gets glossed over in these discussions is that the postmortem needs to be good to be helpful, and in order to be really good it has to be something you plan for from line zero. On a project whose lifespan is measured in years, it’s difficult — if not downright impossible — to remember back far enough to understand the how and the why of some of the earliest design or implementation decisions. This is detrimental to the quality of the overall postmortem; some of the most important decisions are those made early on, because they become the hardest to change and thus have the most impact on the project.
It is therefore helpful for developers to document their projects. Obviously, I don’t mean documentation in the sense of API reference and code comments — that is an orthogonal issue. What I’m talking about is more like a journal. Notes about a decision that was made and the reasoning behind it, comments about ramifactions of particular choices that you did not predict, and so on. This is good stuff to be keeping track of, it can be useful even during development when you need to remember why you opted to implement something in a particular fashion that is now causing problems. It can help to avoid regression of serious flaws (tests, of course, help with this too, but often the flaws exposed by these sorts of changes are conceptual or cross-cut many different subsystems and so might not be caught by unit or functional tests).
In particular, with the growing trend towards more nimble, Agile or Agile-like methodologies, decisions are often made much quicker and with much less bureaucratic overhead. Say what you like about bureaucracy, but it does tend to engender an extensive paper trail about any kind of change. More modern development practices tend to eschew that, which is usually good, but its important to keep track of the things that do matter, otherwise you may find yourself coded into a corner, unaware of how you got there, and repeating mistakes to get yourself back out.
Now, maintaining this development journal might sound like a daunting task that has the potential to consume a huge chunk of an engineer’s time — and that can be true. It doesn’t need to be, though. You don’t need to spend a lot of time writing anything particularly eloquent or well-formatted, you just need to be taking running notes about how things are going. As long as you can read those notes and comprehend them later, you’re good. I leave a Word document on my desktop at work in which I will periodically make notes about problems I observe during the day, as well as comments on what aspects of the project are going really well.
I’ve been able to refer back to this document when my co-workers and I can’t recall why we did something a certain way. Granted, there are Perforce logs and the like, but those usually document only the what of a change, and not the why, which is the important thing. Keeping the journal has been helpful so far, and will be invaluable in compiling a postmortem report at the end of the project and understanding what we should do differently next time, and what worked really well: it will let us know instead of making us guess.
I have much less documentation for SlimDX, which means it’s been much harder for me to dig up the details about some of the more interesting bugs we’ve solved and provide that information in the “making of SlimDX” presentation I’ve been working on in my spare time. I’ve had to resort to having MaulingMonkey from #gamedev grep his massive IRC logs to provide me an archive of every conversation about SlimDX from that channel, and then I had to read them all to find the ones in which the SlimDX team discussed the problems in question, and work from there. It was not a fun process, and what I took away from it is that it would have been much easier to compile the relevant details had we been more proactively keeping track of these things.