Agile on Overdrive

It may sound old-fashioned and laughable these days when techniques like UML are so unsexy, but I can still remember the days back at IBM T.J. Watson in the early 2000's when we were working on a prototype of a new VoiceXML browser to inform IBM's standardization efforts in the space. (The language was supposed to be VoiceXML 3.0, and we abbreviated the research project to "V-3" - pronounced "fau drai" in reference to the German V-2 rocket.) We took VoiceXML 2.0 as a foundation but added full DOM and DOM Events support together with XForms as the data model representation.

It was shortly after IBM's acquisition of Rational Software, so we took advantage of that deal and got our hands dirty with Rational Rose. While one of our colleagues kept pushing us to start coding and iterate towards the result in the agile fashion, given the incomplete specs and many open questions stemming from the combination of so many complex technologies in one piece, we prevailed and kept debugging the design in UML for roughly six weeks. It helped us to uncover and resolve many ambiguities and arrive at the detailed understanding of expected behavior before we wrote a single line of code. Then we sat down and wrote the complete V3 prototype in about three weeks. Afterward, we just fixed few remaining bugs here and there during the evaluation, and we were done - on time according to the original schedule as the whole V-3 prototype work was expected to take roughly 3 months.

I am not going to dispute the overall positive benefits of using iterative software development methodologies, particularly agile methodologies such as Scrum and Kanban. I hope nobody wants to go back to the software waterfalls, at least I don't. That being said, I still want to highlight some problems faced by the R&D teams implementing agile in naïve or kind of fundamentalist fashion.

The Problem of Product Increment

The concept of product increment - an artifact of each and every product iteration - is the cornerstone of Scrum and its kin. It is a great idea, having a presentable and deployable outcome after each iteration. But what to do when there is no product in the first place? What if you are in a green field development situation? What if you are re-architecting and refactoring a substantial portion of your application? To paraphrase Peter Thiel's idea from his book Zero to One: what if we want to create something where there is nothing?

The agile iterations don't help with that very much because they tend to be too short to build an entire production foundation in a single iteration. Some techniques such as story mapping remediate this problem by the concept of a scaffolding or a walking skeleton, but even a working skeleton usually takes at least several iterations to build. On top of that, when building a walking skeleton, you need to know in which direction you want it to start walking as soon as it is built - i.e., you need to build a product vision. Unfortunately, agile methodologies won't help you with that.

Agile teams trying to iterate from "ground zero" to the "first floor" often suffer from short-sightedness and endless rework, because each new iteration is expanding the set of requirements so substantially that the foundations need to be redone from scratch each time. To put it simply, iterating blindly from "zero to one" quickly becomes an enormous waste.

Curse of Product Owner

Another great agile tool - the product owner - while being instrumental in building product increments often compounds the problems outlined above. Why do I think so? Because the role of the product owner is sometimes giving too much comfort to the engineering teams - the PO represents the customer and has the domain knowledge, so more junior teams can have a tendency to completely outsource the domain knowledge to the PO and just blindly implement the requirements as they come. "This was not in the original set of requirements, we have to refactor the database schema." is what you can often hear at later stages of product development, when the engineering team fails to build deep domain knowledge and with it the ability to generalize away from the PO's requirements and anticipate the additional use cases which are naturally part of the problem domain but were not explicitly stated by the PO in the original requirements.

Outside of the Box

So according to my experience, a successful strategy to climb to the first floor from "ground zero" is to assemble a small team of smartest people you have on staff and let them formulate the vision, layout the architectural foundations and devise a longer-term implementation plan outside of the regular iterative development process. There are no shortcuts, and no amount of iterations can replace the deep understanding of the problem domain and holistic approach to architecture and data modeling, addressing the entire classes of domain's use cases rather than a few granular user stories taken out from the top of the backlog. So don't be so "keyboard happy" - eagerly starting coding before taking time to think.

To be continued...