The Origins of Agile

If your engineering team hasn’t already moved to some form of Agile methods (like Scrum or XP), then it’s likely they’re at least considering it. Agile really does attack some key problems that have plagued software teams for decades. But many product managers and designers, and to a lesser extent QA staff, are initially confused by Agile, unsure of their role in these methods. To be clear, these methods absolutely require these roles, but I attribute the confusion to the origin of Agile methods, and I’ve found that when I explain the origins, it helps to illuminate the problems that Agile was designed to solve, and what challenges remain.

Many are surprised to learn that Scrum, the most popular of the Agile methods, is now over 20 years old. It was created in 1986 in Japan. (Yet another example of just how long it can take for a new idea to reach the tipping point).

But most importantly, these methods originated in the custom software world.

The custom software world – building special purpose software for specific customers – has long been a brutally difficult type of software. This is partly because customers notoriously don’t know what they want, but they have a need so they write a contract with a custom software supplier, or sit down with their internal IT folks, who then work to deliver, and when they do, the customer invariably responds that this isn’t really what they had in mind, so the cycle continues and frustration mounts. But the core need still exists, so this has provided job security for countless IT developers, custom software shops, and professional services businesses.

Further, custom software has long been on the short end of the stick when it comes to recruiting and retaining top software talent. Partly this is because many top software professionals prefer to work for companies that are in the business of creating software for thousands if not millions of customers. Partly it’s because software professionals get paid more at product software companies where the product team is responsible for coming up with software products that please many people, or they don’t make money. So these companies know they must hire the talent necessary to create winning products, and they pay accordingly. But to put this in perspective, only a relatively small percentage of software people actually work on commercial product software; most work on custom software.

In the custom software model, since the customer believes he knows what he needs, you’ll rarely find the role of the product manager. Likewise, you’ll almost never find user experience designers. The reasons for this are more complex, and involve a degree of ignorance (relatively few in the custom software world realize what user experience designers do and why they’re needed), and cost-sensitivity (cut costs by letting the developers design), but to be fair, due to the shortage of designers in our industry, the few that are available are immediately grabbed by the product companies that realize how critical they are, and so custom software teams can rarely find designers even if they have the leadership that realizes they need them. Similarly, QA as a discipline is rarely found in custom software projects; again, the developers are typically expected to do the required testing.

Another crucial element in understanding the custom software world is that the vast majority of custom software projects are relatively small and done to support the internal operations of a company – applications like HR, billing, and manufacturing, where the limited number of users means that issues such as scalability and performance are usually less critical.

Historically the custom software world used the Waterfall process because the various stakeholders needed a way to monitor progress during the long process of creating these contract applications. In fact, the Waterfall methods originated here as well.

In the product software world, where the software must sell on its merits, we introduced the roles of product managers to gather and represent the needs of a wide range of customers; designers to create usable and desirable user experiences; and QA testers to ensure the software worked as advertised in the range of customer environments.

But in the custom software world, the same fundamental issues of coming up with something that satisfied the customer continued. For these teams especially, the Agile methods represent significant improvements. They improve the communication between the customer and the engineers. They significantly reduce the risk by building smaller, more frequent iterations so that the customer can learn whether he really likes something or not much sooner, rather than wait for the end of a long process. They help introduce some modern software testing concepts, and they help relieve the team from spending countless hours preparing documents that are rarely read and quickly obsolete.

In general, these are great benefits for product software teams as well, but I always explain that a few adjustments are required. I’ve written earlier about these topics, such as how to insert user experience design into the process, and how to manage releases and deployments, but another area that has struggled is the architectural design area.

The Agile methods encourage engineers to not get attached to their implementation believing that things can be re-factored or re-architected relatively quickly and easily. This is true for the vast majority of custom software, but for many product software systems, such as large-scale consumer internet services which must support hundreds of thousands if not millions of users, this approach can be naïve.

So it shouldn’t be a big surprise that the main issues that many product software teams have encountered with Agile methods stem from the custom software origin. The many books, articles, and training classes that don’t mention product managers or any form of user experience designers (interaction designers and visual designers) weren’t meant for product software teams.

My suggestion to teams moving to Agile is to make sure the firm you hire to help your organization actually understands the differences that product software demands. Most don’t, but enough do.