When to employ Agile and when not to. Can Agile and Waterfall work together?

Introduction

Rate how much your business processes, organizational structure and culture help you develop and deliver the product(s) or service(s) your organization is working on, using a scale from 1 to 10, where 1 is fully dysfunctional and 10 is everything as good as it can be. If you believe they hinder your business, you can also assign negative values.

I’ve seen numerous times in various industries that companies of all sizes try to blindly or mechanically implement a delivery/project management framework or methodology to improve their capabilities (e.g. PRINCE2, Scrum, DevOps, etc.). Most of the times they were disappointed with their results.
This is especially true when it comes to Agile frameworks, methods and practices. Companies see the extraordinary results some organizations have achieved using Agile and want those results themselves.

Frameworks are pre-canned solutions. We can get so excited about them that we can lose sight of the real business problems and goals we have and what we need to change to get better at solving those.

These series of articles will help you understand the type of business problems you may encounter in your every day work life and will help you find the optimal organizational structure, processes and cultures for solving those problems.

Problems tend to differ in key attributes and characteristics and these characteristics require us to use very different strategies to successfully solve them.

Most of us are not aware of the fundamental differences between the nature of problems so in solving them, we revert to a strategy that is most familiar to us, instead of using the one which works best.

We will explore one of the key characteristics of problems: complexity, and how different complexities require very different strategies, organizational structures, processes and mindsets to successfully solve the underlying business problem.

This article is the 1st installment in a series of two articles focusing on understanding different problem complexities and learning to select the optimal strategy for solving a specific problem type.

How to read this article

This article uses evidence based coaching exercises to help you come up with your own views, generate insights and reflect on your own situation. Although it contains a lot of useful information about organizational design, the article was not designed to be solely read or skimmed through. It has multiple short exercises which, together with the information in the text, can help you reach deeper insights and generate useful actions.
Skipping any of the exercises in this article will eliminate a huge portion of the value that you could take away. It is highly recommended that you do each exercise as you encounter them and before continuing with the article.

Background reading

There are lots of people who have studied and categorized the complexities of problems but I would particularly recommend the following books by Ralph D. Stacey: “Strategic Management and Organisational Dynamics (7th edition)”, “Complexity and Organizational Reality: Uncertainty and the Need to Rethink Management after the Collapse of Investment Capitalism”

Here is a short summary of one of his works, the Stacey Matrix:
https://www.gp-training.net/training/communication_skills/consultation/equipoise/complexity/stacey.htm

His work has been adapted to software development as well:
https://www.scrum-tips.com/2016/02/17/stacey-complexity-model/

I’d also recommend the book “The Agile Mind-Set: Making Agile Processes Work” from Gil Broza. Despite the title it’s not just about Agile, it provides a good comparison of how different problems require different assumptions, approaches and values to successfully tackle them.
Gil does a great job in explaining the connection between mindset and the underlying problems and some parts of this article build on top of his work.

I’ve worked out my own way of classifying problems based on my experience and the work of others like Broza or Stacey. I’ll present my own views here. There are many other ways of classifying problems either based on their complexity or something else. Although the model I will present has inaccuracies (and I’m well aware of many of those), I’ve found it very useful in practice. It’s also quite simple, so hopefully it will be easy to learn compared to other models.

What are the different complexities problems can have? How can I differentiate between them?
What are the successful strategies for solving problems with different complexities?

I. “Simple” Problems

Simple Problems are the ones where you know what you are trying to do and you also know how to do it.
There are only what we’ll call ‘known knowns’.
Typically these are tasks that you have either done or plan to do a lot of times, without substantial change in the execution of the task. E.g. brushing your teeth.

For these problems it’s also reasonable to assume that the customer knows what (s)he wants, (s)he understands the solution to his/her problem and not just what the problem is.
E.g. your cupboard is old and worn out and you decide to replace it. You know the reason for the purchase (i.e. the ‘problem’), the size of the cupboard, the colour your want and you remember the nice corner store where you purchased the current one so you also know the place where will you buy the new one.

What’s the strategy to solve “Simple” Problems?

Exercise 1

Now think of three separate examples of simple problems that you have solved. Remember these are problems with known knowns only, problems where you knew the problem and the solution as well and neither of them changed. Take your time.
Now, try to answer the following questions:

  • If you had to articulate the strategy you used to solve these problems or a pattern in your strategy, what would that be?
  • To what extent were you following a predetermined, relatively well-defined set of rules, processes or a plan that you (or someone else) created?
  • Did you follow this plan almost automatically or semi-automatically?

Exercise 2

  • Can you enumerate the steps you do to prepare coffee or tea for yourself?
    Are you thinking about these steps while you are preparing your coffee or tea or have these steps become so ingrained through practice that you just follow a sequence of small actions without thinking (e.g. boiling water)?
  • Do you brush the upper or the lower part of your teeth first? Is it always the same or do you vary it? Do you choose this consciously or is it habitual? How frequently do you think about or change the strategy you use to brush your teeth?

We solve Simple Problems by carefully enumerating a sequence of activities, fine tuning them and practicing them to perfection.

Another example is fast food restaurants where everything is standardized, from the size of your burger to the time the meat spends in the oven or what steps employees have to follow when they serve a customer. Even managing conflict is semi-standardized, there are guidelines or policies about when to offer a substitution meal, etc.

Create a plan by measuring and analyzing the environment and then practice it to the point that it becomes second nature, then execute it over and over again. This strategy works best for Simple Problems.

This is the domain of classic Waterfall planning, standardization and Taylorism.

II. Complicated Problems

Complicated Problems have some uncertainty: either the requirements (what you are trying to achieve) are not entirely clear or the way to fulfil those requirements (how you will do it) is not clear.
However, you are aware of what you don’t know. There are what we’ll call ‘known uknowns’ in your problem.

Example

Let’s say you go home by car, you know what steps to take, which roads to follow, where to turn left, where to turn right, etc. You also know that today the police have closed some of the roads on your fastest route because someone important is visiting your town but you don’t know exactly which roads are closed. In order to figure out a viable alternative route you know that you need to investigate exactly which roads are closed. You know what you don’t know.

What’s the strategy to solve Complicated Problems?

Exercise

Now think of three examples of Complicated Problems in your life. Remember these are problems with known unknowns (as well as known knowns of course).

  • If you had to articulate your strategy in solving these problems, what would that be?
  • How does this strategy differ from the strategy you have employed in solving Simple Problems?

The basic strategy is the same as the one we had for Simple Problems: create a plan and execute it, but there is a difference here. This time the plan will have activities for investigating, discovering and learning unknown things.

E.g. you might plan to open the route planner on your mobile phone to check out which roads are not closed. This takes time and effort and the learning from this activity modifies your original plan of how to go home.
This activity may also lead to an unexpected answer like that all routes are closed or that you can’t go by car and need to use an alternative transport mode.

These uncertainties give rise to classic project management methods and practices (e.g. PRINCE2, PMI PMBoK).
A few examples:

  • A longer analysis phase at the beginning of the project to eliminate or reduce uncertainties.
  • Buffering or implementing ‘Change Budgets’: as some activities are exploratory they might reveal unexpected activities that will prolong the original plan or increase its cost, this needs to be accounted for
  • Risk management: there are known events that might happen and if they do there is a known (or estimate-able) impact they will have on the desired outcome or the delivery. However, we don’t know if they will actually happen (known unknowns).
  • Having to monitor progress against the plan and readjust the plan when necessary: as not everything is known or under our control, it might be necessary to change the scope, the schedule, the cost of a project or sacrifice some of the value that we would have wanted to deliver. We rarely have to do this with Simple Problems. How frequently do you need to skip one tooth during your morning teeth brushing?

So we still create Waterfall Plans but these are much more sophisticated vehicles than the ones we used for Simple Problems. These plans include change management, risk management, buffers, frequent checkpoints for monitoring and processes for adjusting, if necessary.

Some examples of Complicated Problems are replacing a component in a piece of machinery (e.g. a car) or rolling out a new piece of hardware to tens of thousands of employees or customers.

III. Complex Problems

This one is tricky. These are problems with unknown unknowns: you are not aware of key information about the requirements or the way you will fulfil those requirements and you don’t even know that you should be aware of that key information. You cannot proactively seek out, learn, investigate or analyze the missing information.

Many times this is driven by the fact that the customer doesn’t know what (s)he wants. (S)he is capable of understanding her problem but not what the solution would look like.

Examples

Software development: you don’t know that the technology you are using will not integrate with another technology that is also being used in a different part of the same software. You find this out later when you try to make the two technologies work together, it fails and you have to rewrite significant part of your code.

Facilitation, relationships: you work with a team to help them improve their decision making capabilities but you are not aware of the past history between two team members. An unresolved and unspoken conflict they have had in the past is having a detrimental impact on the entire team’s capability to make decisions and agree on things. You are not aware of this at the beginning, so you just start to employ your normal training and coaching plan which proves to be ineffective during the first few weeks.

Where do Complex Problems occur?

If you work in any of the following fields you will probably find it easy to find Complex Problems in your work life: software development, data science, medical research, facilitation, coaching, psychotherapy, Special Forces, meteorology, etc

If not, try to look for quarrels you’ve had with people whom are very close to you and maybe you can find an example there where you were not aware of all the emotions, thoughts or memories which were in play and you didn’t even know that you should have been aware of them.

What’s the strategy to solve Complex Problems?

Exercise

Now think of three examples of Complex Problems in your life.
If you had to articulate your strategy in solving these problems, what would that be? Try to answer the following questions about this strategy. If you find it difficult to answer any of the following questions, then try to imagine what would you do if there were no rules, policies and existing practices in your organization. What if you could start from scratch? What answers would you give to these questions if you were the CEO of your company?

  • How much planning does your strategy have?
  • What is the difference between the planning in this strategy and the planning you employed in the strategy to solve Complicated Problems?
  • What is the proportion of detailed activities in these plans compared to outcome-focused high-level activities that only give direction? (E.g. write an algorithm that checks if the date entered is valid (specific activity) vs. need to learn if technology ‘z’ is fit for purpose.)
  • What percentage of the activities (and their estimates) in these plans survive until the end of the delivery? How frequently do you need to amend this plan?
  • If you find you need to replace or amend activities in your plan, on average how much time elapses between starting the activity and recognizing that it needs to be amended?
  • What proportion of your activities in your plan are about experimentation, discovery or trial-and-error learning compared to activities which are about delivering something fixed to the end customer?
  • What is the level of detail with which you can reliably plan activities for the next few days? What about the next few weeks? Next few months? Give these percentage numbers.
  • If you compare the level of detail with which you can reliably plan activities, is there a discrepancy between near future and far future activities?
  • If there is a discrepancy, is this consistent across your projects/work or does it vary? What patterns do you see?

A metaphor for solving Complex Problems

Metaphorically speaking, solving a Complex Problem is like being in a dark forest with a single lantern, without a map, and having to find your way out.

It’s pitch black, you have been in a lot of forests but not in this particular one. You have no maps.

All you have is a small lantern which illuminates a small circle around you. You can see what’s immediately ahead of you with great detail but when you try to move your gaze to the distant the details fade away quickly, everything is covered in darkness and fog (i.e. uncertainty). You hear some noise from the forest but it’s hard to rely on this to navigate. Some things also move in the forest, the sound of a wild animal you heard from the left 2 minutes ago is now on the right (the market changes).
You know the next step but you don’t know what the next step will reveal (unknown unknowns). You seem to be on a path now but will that path continue or will it end abruptly? Will you find a poisonous snake next, or a dangerous predator?
Maybe there’s a wall or a hill in the way and you will have to turn back or walk around in frustration because you have lost so much time. Maybe you’ll find a herb or something to eat. Or maybe there is a shortcut and you find your way out quicker than expected.

Could you plan your way out of this forest?

If Planning and standardization fail miserably when trying to solve Complex Problems, what works?

As you probably suspect by now, planning is an inappropriate tool for solving Complex Problems as you cannot plan an activity without awareness of it.

The alternative is Emergence and Evolution: devising short activities based on what we do know now with the intention to learn more about the requirements and/or about how we will fulfil those requirements (e.g. learn about a new technology or the speed by which we can deliver).

The only way to reliably learn about the requirements is to deliver some value to the customer quickly and get his feedback.

Had the customer known more about the requirements before our product existed, he would have articulated them already. He probably struggles to articulate his problem or understand the solution to his problem as much as we do.

“If I had asked people what they wanted, they would have said faster horses.”

Henry Ford

Up front analysis in this case is like sitting in the dark forest and speculating about the rock/animal that might be 10 steps away, or might not, maybe it’s a lake instead or a path. We don’t know.

If you work in software development, you probably know that the most successful start-ups get something to the hands of their customers early and frequently. There isn’t much of a difference between development and production as they are delivering live productionized software from day 1.

So as we deliver small pieces of value to our customer some of the unknown unknowns will reveal themselves. We may become aware that we don’t know something about our customer, requirements, product, technology or our capability to deliver. We will learn (hopefully) the importance of this information as well.
Maybe we’ll get lucky and we’ll learn precisely how our customers want a certain feature within our product to work (known knowns).

We use this learning to devise a new set of activities, to make the next step in the dark forest. We may change our direction or pace based on this learning.
We may have reduced the complexity of some parts of the problem we are dealing with so we may use our learning to create small plans for a subset of our deliverables. We break down the work and reduce its complexity.

Failure is an inherent and unavoidable part of solving Complex Problems

Sometimes we’re unlucky and we learn that no one wants the feature we have just developed. This can be especially frustrating if customers told us they’d like it but once they use it in real life, they realize it’s no good for them. It’s great to find this out early and quickly. Due to the nature of the problem we are solving, we couldn’t have known this in any other way but by trying and failing.

We need to fail fast and celebrate failure instead of punishing it if we want to solve Complex Problems.

Yes, it is painful to encounter a wall in the dark forest and it’s painful to learn that we have to walk around it and lose time. However, remember, your problem is unique, no one has solved your exact problem before. There was no other way to learn this but to explore and fail. If we want to optimize our delivery we need to shift our focus from avoiding failure to making it faster.

Solving Complex Problems requires Emergence and Evolution which is an iterative and incremental value delivery model based on devising short value delivering activities (experiments), usually a few days each, with the intention of learning about the requirements and/or how we will fulfil those requirements.

This is the domain of Agile methodologies, frameworks and practices, e.g. Scrum, eXtreme Programming, Crystal, etc.

IV. Complex Adaptive Problems

There is a special type of Complex Problem, Complex Adaptive. This is a problem where what we need to do or how we need to do it changes when we interact with the problem. You become a player in a game you are observing. E.g. just the act of delivering value may change the needs of the customer.
Metaphorically Complex Adaptive problems are Complex problems on steroids.

Some example domains are coaching, psychotherapy and some parts of software development.

The fundamental strategy to successfully tackle these problems is the same as for Complex problems but these problems are more unpredictable and every step is more challenging, just a few examples:

  • You may find that you need faster iterations in your emergence and evolution cycle
  • You will need an extremely flexible mindset to change direction, even more flexible than the one you needed for Complex problems.
  • Your honesty, transparency and capability to reflect on very uncomfortable and unforeseen developments will be tested more than ever.
  • Although you do need psychologically safe environments where people are encouraged to speak up and proactively address issues in order to solve Complex problems, when solving Complex Adaptive Problems you cannot allow even occasional instances of oppression or control of sharing of relevant information.
  • Customer development and your capability to retest your assumptions about the requirements will be tested. You may find it necessary to automate certain elements of this task as you will have to do it frequently.

V. Chaotic Problems

Chaotic Problems have an extremely high proportion of unknown unknowns compared to Complex Problems. They also have known knowns which are true now but will become false at some point in the future without notice or warning. In other words, you think something is true but it is not.

Good examples are natural catastrophes, an oil rig on fire or a riot that became turbulent and spiralled out of control.

We will only visit this problem type briefly as most businesses cannot be successful in a sustainable way if they deal only with Chaotic Problems.

What’s the strategy to solve Chaotic Problems?

As information that is true now might be invalidated at any point in the future even Emergence and Evolution doesn’t work. Planning is also ineffective as all forms of planning are based on a good understanding of a stable reality.

Chaotic Problems are super unpredictable, turbulent by their very nature and cannot be solved directly. The strategy that works is to stabilize the situation, reduce its complexity to Complex or Complicated and pick-it up from there using the other methods discussed above.

The winning strategy is:

  • Increase interaction between individuals and other parts of the system (e.g. technology).
    The goal is to notice as quickly as possible when something that was true becomes false.
  • Act. Don’t spend too much time thinking or analyzing as thinking is planning and planning is mostly useless here. Do something that you reasonably believe will stabilize the situation (e.g. use the fire extinguisher).
  • Very quickly reflect on your actions: if they have reduced the complexity, then continue, otherwise try something else.
    There is no time for blame, over thinking or bureaucracy. Within the boundaries you just have to do what stabilizes the situation.
  • The anti-pattern here is trying to arrive at a desired Complex or Complicated end state instead of trying to arrive at any one of them. It is not possible to control or predict the outcome of Chaotic Problems. Quickly stabilizing to any Complex or Complicated state and then improving from there is better than staying in the Chaotic domain longer. Hoping that we can exit the Chaotic state into a particular Complex state which is more desirable than another Complex state can lead to further damage to the business.

Read the second part of the article here

This article is the 1st installment in a series of two articles focusing on understanding different problem complexities and learning to select the optimal strategy for solving a specific problem type.

You can read the second part here, where we will continue by focusing on the different organizational structures, cultures and business processes necessary to solve different types of problems.

Do you need help? Get in touch.

I’m using professional coaching and evidence based methods from the latest research within psychology to help my clients with the behaviour change they need to achieve their goals so that leaders and organizations can identify and grow the optimal organizational design and make the hard decisions which work in their context.
If you need help with any of these, please don’t hesitate to contact me, I’m here to help:

Contact Me

Also please subscribe if you liked this article and share it on social media: