Agile Management in a Waterfall World

An entrepreneur comes to a venture capital firm with the design for a game-changing electric vehicle. The VC wants to know: if we give you capital, how many of these great new cars will you produce in 3 years? The entrepreneur does not yet have a factory or workers or suppliers. He does not yet have regulatory approvals or distribution networks. What he has is an innovative design and the belief that, given the chance to launch his EV business, he will be able to build and ship, solving challenges as they arise.

This example presents the fundamental tension between sources and uses of capital. Investors and clients want clear timelines and deliverables. The more complex and resource-intensive the projects, the more capital at risk, the greater the requirements for waterfall planning. When an investor or client puts up capital, they expect to know how it will be used and when they can expect a result. It is only rational.

On the other side of the spectrum, the entrepreneurs and vendors realize that the more complex and resource-intensive the projects, the more agility is required for a successful outcome. Designs, suppliers, customer sentiment, underlying technologies, and regulatory requirements are all subject to change. To get to the right result, it is important to iterate and adjust.

Consequently, “waterfall” has become something of a dirty word in tech circles, made the dirtier by the fact that its practice is far more pervasive that many engineers are willing to admit. Two decades after the introduction of “agile”, let us unabashedly state a shocking truth: there is nothing dirty about waterfall — it is simply a reflection of companies’ need to plan. In dealing with multi-million dollar budgets, investor obligations, and client requirements, the need to plan will always be part of the equation.

The Journey of a Thousand Miles

Many of us have been conditioned to think of waterfall and agile as mutually exclusive methodologies. It is time we understand that they are simply sides of the same coin. Waterfall maps us to a general destination. Agile gets us there, one step at a time.

If I am going to bicycle from Los Angeles to Niagara Falls, I can make some reasonable estimations. I can plan a route and approximate the general timeframe, give or take a week. I can assess how much money I may need, making assumptions about the costs of a new bike and supplies (CapEx) and the costs of things I will need on the way like food, lodging, and repairs (OpEx). This is waterfall planning, and it will give not only myself, but my friends and family a sense for how much money they may need to lend me and when to book their plane to Niagara Falls to welcome my victorious arrival. It is a rational way to plan ahead.

Once I start my trip, I am going to discover new information every day that will impact my original estimates. Everything from unexpected weather, road closures, and flat tires to new sights, social occasions, and festivals thrown in my honor by towns breathlessly following my travel blog. Well, maybe not the latter, but you get the idea. I will have to adjust my ride day to day and these adjustments, in turn, will affect my initial waterfall plan.

There is no question that being agile will give me a better ride. If I am running ahead of schedule, I can take more time to enjoy the sights and experiences along the way. If rain keeps me hunkered down for a whole day, I can double my bike time the next day to make up for lost ground. And if my bike gets stolen and I must wait a whole week to get a replacement shipped, well, I can let my friends and family know that my whole waterfall plan has been pushed out by a week. They will understand (I hope).

Staying Dry Under the Waterfall

While it may be easy for me to explain to family and friends my bike ride delays, explaining product ship delays to enterprise customers, with millions of dollars on the line, is a touch more challenging. Thus, the fundamental tension between business, who wants a perfectly detailed multi-year waterfall plan, and engineering, who wants the perfect freedom to iterate, one sprint at a time.

Mediation between business’ waterfall needs and tech’s agile habits is not merely advisable but required for organizational success. The solution? A horizontal project management function that can plan and track the waterfall while keeping the engineers dry under the twin umbrellas of agility and accountability.

This is easier said than done. Project management is the hydra of the technology organization. If you have seen project management at nine different companies, you have seen nine different ways of doing project management. The nuances are myriad. Some project managers live inside operational teams, some live outside. Some project managers use old school spreadsheets, some use bleeding edge cloud tools (more on this later). Some project managers are certified PMPs, Scrum Masters and Six Sigma Black Belts, and some are just folks with experience.

Go far enough down the tributaries of project management organizational structures, reporting arrangements, methodologies, tools, certifications, and you are bound to throw up your hands in exasperation. With all this project management complexity, no wonder we struggle to mediate between business needs and tech deliverables. How can we solve translating waterfall to agile and back again when none of our translators speak the same language?

Simplifying the Roadmap

On the river toward success, beware the tributaries leading to nowhere. How can we tell the project management river from the tributary? Ascend to a higher elevation and see a bigger chunk of the landscape. In our context, that means understanding the underlying relationship between waterfall planning and agile delivery and capturing the basics of mediating the two.

Remarkably, the relationship between waterfall and agile is so simple that you already know it. If you have ever drawn up a shopping list, congratulate yourself. You have done project management. Let us consider that example for a moment.

Preparing a grocery list is waterfall planning. You plan for your upcoming meals, snacks, and drinks. You consult recipes and social schedules. You assess the volumes of your pots and pans. You consider the needs of your family or roommates, who are your customers when it comes to the grocery list.

When you hit the store, your waterfall plan hits an agile reality. You pick and choose varieties of produce. You consider the pros and cons of vegetarian meats and gluten-free breads. You pick replacements for out-of-stock items. Sometimes you adjust the selected ingredients to fit your mental meal plan and other times you adjust your mental meal plan to fit the available ingredients. The entire time the little project manager that lives in your mind is busily mediating between your waterfall list and your agile shopping.

Cycling Toward Commercialization

Remember those old pictures of early bicycles? They had a tiny back wheel and a massive front wheel that looked like it could fit about twenty of those back wheels within its circumference. The inventors did not realize it at the time but along with a revolution in locomotion, they were creating a powerful metaphor for agile project management in a waterfall world.

The big wheel is the waterfall plan — it is the grocery list for our meals that ensures our expenditure of capital will provide the means for satisfying our appetites. It is the thing we are implementing — the product, infrastructure, or feature set to address our customers’ needs.

The little wheel is the agile delivery — it is the ubiquitous cycle of plan, design, code, test, release, review that we live and breathe in the world of engineering. The thing is, when it comes to planning our roadmap, we do not need to know exactly how each turn of the little wheel will go. All we need to know is the approximate number of little wheel rotations required to make the big wheel turn once.

In other words, we must segment our waterfall project plan into reasonably and similarly sized agile cycles, then figure out how many we can run in parallel (taking into account resource availability and dependencies), and voila — we have a reasonable approximation of a project plan.

Considering the Complications

Is this an oversimplification? For large, complex projects it is difficult to know all the deliverables upfront, it is impossible to figure out all the dependencies, and it is unfeasible to project the technical challenges that will arise. Not to mention the vagaries of ever shifting client needs, operational priorities, technological advances, talent availability, and budgetary constraints.

It is understood that when it comes to large-scale project planning none of us has a crystal ball. At best we have a cracked tin ball of our fragmented experiences held together with the fraying rubber bands of our best guesstimates. The good news is that a crystal ball is not required.

When we take our big wheel/little wheel project management bicycle on the road, we need not know the terrain in advance. So long as we stay in balance, meaning our approach is consistent, we can make turns and adapt to changing conditions.

Our agile cycles for individual deliverables give us the flexibility to adjust to the small bumps in the road while keeping the deliverables themselves small enough to drive transparency and accountability. At the same time, when we encounter larger zigs and zags in the road of reality, we can add or remove cycles from the big wheel to keep our project pointed at the larger objective.

Building the Frame

Do we need the big wheel and the small wheel both? If there is one thing we have learned from decades in the trenches of the information age, it is that waterfalls result in clunky, outdated and unfriendly products while total agility often leads to beautiful features in product-less voids.

More precisely, the problem with a pure waterfall methodology is it is too heavy on the planning and too light on the feedback of reality. The problem with a pure agile methodology is it is too heavy on the detail and too light on the big picture.

Each one is a unicycle — it can work, but it is far too easy to fall off. By taking the best of both worlds, the big picture approach of waterfall and the detailed iteration of agile, we can create a more balanced machine.

The hybrid waterfall/agile bicycle approach can be applied to any technology project using a simple 1–2–3 method. First, chunk up the project into similarly sized segments. Second, overlay the project on a Gantt with known resources and dependencies constraining concurrency. Third, run agile cycles for each segment and adjust the overall project plan as necessary. Although this may sound straightforward, it is far from easy to get it right.

Guideposts for the Road Ahead

If you can effectively apply the hybrid approach, you can deliver technology projects of any complexity. Use the following 4 key guidelines to keep your projects on track.

1. No bells and no whistles unless you are running a bell and whistle factory. Minimum viable product (MVP) is the name of the game for the big wheel. You can add more features later. MVPs stack, like floors in a building, and today’s bells and whistles are tomorrow’s MVP. Let tomorrow’s rotation of the big wheel deal with those, for today stick to what you know for certain must be built. This will shorten the project plan and shorter project plans are easier, faster, and less costly.

2. Do NOT overstuff your building blocks. Keep the deliverables — the little wheels — inside your project plan around the same size. That will make it easy to add, remove and substitute. When your bricks are the same size, it is also easier to track throughput and keep the bricklayers accountable.

3. Every jigsaw piece is equally required. Maintain perspective and consider every deliverable in the context of the big picture. There are going to be many rabbit holes between your starting point and your objective — many voices shouting that this deliverable or that task is more vital than the others. Remain disciplined. The time you are given is to complete the whole picture, not a portion of it, no matter how well. Ignoring rabbit holes is the only way to stay on track.

4. Every team needs a coach. Every deliverable must have an owner. There may be others assigned to do the work, but it is on the owner to ensure the work gets done. The project manager owns the big wheel and deliverable owners own each of the small wheels. This creates a chain of accountability that keeps the whole bicycle moving forward.

An Afterword on Tools

The world of cloud tools for project management has exploded over the past decade. Beyond the gray beard of Microsoft Project, we see Zoho, Asana, Smartsheet, Monday, Wrike, LiquidPlanner, and Jira, just to name a few.

Is effective project management just a question of selecting the right tool? Absolutely not. You can have exceptional project management run on Google Sheets and singularly ineffective project management run on the latest and most popular cloud tool.

Never try to build a process around a tool. That is putting the cart before the horse and expecting the horse to run. Rather, select and configure a tool around your process. If you have a well-oiled process, the right tool can add synergies. But if you have a poorly designed process, the tool is irrelevant. If you plot the wrong course, it does not matter how fast and powerful your bicycle.

The question onion applies. Start with why you are building the project, the problem you are solving. Then proceed to how you will build the project, the project management methodology. Finally, determine what you will build — the milestones and deliverables. The final piece is merely an extension of the what, more precisely: what will you use to track and manage the deliverables.

I have devoted much of my career to solving complex challenges through innovation across management, technology, and process.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store