The Essence of Agile Product Development Explained

Debunking the spiral of ever increasing value

The word “agile” has become somewhat of a controversial term in many groups. While many organizations and teams claim to be “agile”, in reality few are. This has caused the essence of agility to be diluted, not because it isn’t as valuable as it promises to be, but because it is misunderstood so easily.

The journey to agility in just 1, 000 steps and 10 years

Becoming agile is a journey. It’s not a switch that you can simply turn on. It takes time, and it takes a lot of effort. Mostly, it takes opening yourself up to try and learn a lot of new things. And fail often.

When I first came across the world of quality — which is at the heart of the Scrum process design as I’ll explain later — it took me the best of 2 years of learning and teaching it to others to fully understand the concept of quality and the underlying work of Deming (in which I’m still not an expert to this day). Little did I know how important this foundation would become later in my career. At that stage, the word Agile or Scrum didn’t exist mainstream yet.

The closest concept to an agile coach was Quality Managers — responsible to align organizations, customer satisfaction and continuous improvement. Now the label is different and the focus usually on fewer teams, but the ultimate goal is the same.

When I came across the Theory of Constraints —a process to identify and support those things that stop you from being productive — it again took years to master the principles (and I'm far from being a master in the field).

Psychology, human behavior, change and habits are subjects I’ve actively been studying for more than 5 years and each day I realize exactly how little I know. It is, however, the most fundamental goal of agility. The process of becoming more agile in what you are doing is the process of changing harmful habits and behaviors into more healthy ones.

Coaching and the art of asking the right questions to guide a person towards an answer rather than telling it was one of the hardest skills I’ve had to develop. It asks you to unlearn everything management has told you a good ‘manager’ should do for the past 100 years. It definitely doesn’t take a few days or even a few weeks or even months to develop and it takes a lot of actual practice, something I practice whenever I get the opportunity.

Yet, people call themselves agile coaches when most are unable to pose any questions when dealing with teams, so used to giving instructions.

Understandable.

Asking doesn’t come natural to us. Traditionally, the manager or specialist was the one who had the most experience and they saw their role as telling people how to do something. Asking for help was seen as a weakness, while in an agile world it is your single biggest strength.

Becoming agile, similarly to my personal journey, is an ongoing learning journey. It doesn’t stop. It evolves. It grows. It expands.

So.

Let’s talk about agile…

The first mistake most people (including me) make — sometimes out of laziness (as in in my case) or out of ignorance is using the word to describe something vastly bigger than what the word actually means. The word “agile” has been changed from the intended adjective to describe how software (and specifically software) is developed, as originally included in the manifesto for agile software development, into a proper noun “Agile” — to refer to teams, organizations and even tools. As a result the meaning of the word has gotten lost in translation.

The meaning of the word agile has gotten lost in translation.

Where the intent of the manifesto was to serve as north star or compass to help teams develop software in ways that better responds to changing requirements and needs, it became an abstract term to describe anything you wished to perceive as ‘better’ than other teams, organizations or processes. If you started using Slack after it was previously banned in your organization, suddenly this new felt mere snippet of freedom was considered a sign of organizational agility. If your team had a Scrum board (even if it was only on Jira or Trello) and used sticky notes, people deemed themselves as more ‘agile’ than those mere mortals who didn’t.The more sticky notes, the more agile you were. If they had a backlog and daily stand-ups, they were considered super ‘agile’ in relation to those doomed to long sit-down meetings.

The second mistake commonly found is that people shorten the original manifesto into only the four values which forms the body of a longer document, omitting the introductory and ending sentences which are essential to understanding the context of these values and how to use them.

It is in fact possibly the most important part of the manifesto.

The introductory sentence clearly states that the only way to develop software in a more agile way is to constantly look for better ways to do what you already do. It doesn’t end once you’ve worked through it once. Rather, it advocates for the ability to constantly learn and improve and adapt. It’s intended to be a starting point of a lifelong learning culture, not a one-pot easy recipe for a single meal.

After the four values, which essentially is a decision making guide, the manifesto ends with a simple sentence that states that the items on the right are not wrong as such, the items on the left are just more valuable than the items on the right.

The manifesto is a decision making framework intended to help you develop better judgment.

If presented with a choice, for example, people always trump process and tools, but that doesn’t mean there shouldn’t be any processes or tools. It merely means that the processes and tools shouldn’t dictate the development process or restrict the possibility for people to do their jobs, which in most cases are the truth about tools and processes in organizations.

The essence of agile product development

At the heart of productivity, which I use interchangeably with the word agile as used by most people, is Deming’s — the father of quality (not only software quality) — Plan-Do-Check-Act cycle which is also the essence of the ISO 9001 standard on quality management systems; as well as the Scrum framework.

The underlying concept that dictates quality, according to Deming’s work, is that the faster you can get feedback and respond to that feedback, the higher the chances that you are developing the right product and thus meeting customer expectations. And quality is essentially all about meeting expectations.

Like driving a car, when you fall asleep on a long road the rougher chevron lines will wake you up to enable you to correct yourself and get back on track. Your focus is on staying as close to the goal as possible and in order to do that, you always need to know exactly where you are in relation to where you want to be through feedback and probably more importantly, by having a clear goal.

1. Clear goals

We’ve lost the art of setting meaningful goals, lost in a world guided by measurable KPI’s and data driven analytics. A good goal is rarely a number. A good goal contains the reason why you’re doing it in the first place.

Maybe you’re trying to build trust, maybe you’re trying to connect buyers and sellers, maybe you’re hoping to create the future of finance to enable more people to enjoy wealth.

But it’s not a 10% increase in user onboarding or to complete all the stories you picked from the backlog during planning session with no seemingly connecting golden thread to pull them all together into a more cohesive goal.

To set a goal, ask yourself:

What are you hoping to achieve when you release the software? What is the impact of your product or service? What do you hope people will be able to do?

Without a clear reason or problem to solve, you’re most probably solving the wrong problem, or solving the problem wrong. You simply can’t tell the forest from the trees while you’re in it though, so you won’t notice until it is finally released.

That’s not very agile.

2. Fast feedback

While the goal is essential, as essential is the ability to gain (and react to) faster feedback from actual users. Not the analytics. Not the Product Owner or Director of Marketing who sits in their ivory tower and hasn’t visited an actual customer in months, or the shareholders who pushes for more profit, but people who use your product within their context. If you’re developing medical software, it means visiting the hospital. If you’re developing an AirBnB or Uber, it means talking to and observing hosts and guest, or drivers and passengers.

Everything in agile software development supports this single principle of fast feedback — if done correctly.

Having an on-site customer (or representative) to clarify requirements, or making time to spend with actual users in the field, is the first opportunity to get feedback faster. Rather than spending weeks, months or years gathering requirements and designing the perfect system in your office without windows and isolated from reality, collaboratively identify and work on only the most important parts with the focus ensuring you understood their needs and that what you are doing is helping achieve that goal.

Next is the estimation and backlog grooming sessions where user stories (requirements) are adapted based on feedback from the developers early in the development process. Rather than hand over a perfect story designed in isolation only to find out that technically it isn’t feasible to build, get technical input early on and adapt your designs from the word go. There’s no such thing as perfect in the agile world, so don’t get too attached to your designs or your code. The goal of agility is to maintain equilibrium between the different moving parts in order to maintain harmony, much like a surfer has to ride the waves of the ocean to stay above water.

There’s no such thing as perfect in the agile world, so don’t get too attached.

Pair programming and Test Driven Development (TDD) are the next opportunities for rapid feedback in the process of development, eliminating the need for code reviews as it happens in real-time, while failing tests written as a first step keeps programmers from tumbling down the wrong rabbit hole. It’s kind of like a check-list to make sure nothing is forgotten while getting lost in the code (which is easier than what most people would like to believe), much like the chevron lines on the road keeps you from getting too far off the road when you fall asleep driving.

Next is the review ceremony where small iterations of the product is presented as working software in a demo where the customer can interact with the software and give feedback immediately before it is released. Words are a very unreliable source of judging reality. Telling someone and showing someone are two very different things.

In an agile world, you show - never tell - progress.

3. The spiral vs linear approach

Possibly the hardest concept for most people to understand when it comes to agile software development is that the process is a spiral rather than a straight line, repeating over and over again adding a little bit more value with each iteration.

Traditional project management according to PMBOK sees software development as a project with a defined start and end. It’s a linear road with a known destination. In reality, however, software doesn’t have an end and in most cases the destination is rather fuzzy and keeps changing.

There are no straight lines in nature. Neither in software development.

The software starts, but then it keeps evolving and maintenance is not something that can be handed over to a maintenance team to fix bugs on existing code. Rather, it requires a constant refactoring and changing of features and functions to cater for the changing needs as more and more users start using it. It’s rather like molding and remolding soft clay rather than making one pot burnt in the fire to harden the clay.

The difficulty, however, lies in knowing what the core of each iteration should be. In traditional project management — often compared with building a house — one thing needs to be completed before adding the next layer. Foundations are laid before the main structure of the house is added. The roof and windows can’t be added before the main structure isn’t complete and the electric wiring and plumbing can only be done right at the end.

Software development doesn’t work that way though.

In software development the goal is to automate something a user does to make their use of time more efficient, and they do a lot of complex stuff all at once and in different variations.

Most systems don’t just do one thing though, they need to do many things all at once in order to be valuable. Sometimes even conflicting things and you need to prioritize and choose.

Logging in and onboarding — which can be seen as the foundation of many systems — is not the most valuable thing to develop first. You would rather have a skeleton version of the key features to validate the process as first phase before spending time on onboarding. There’s little value in wasting a lot of time in perfecting onboarding when you don’t have a value proposition that will ensure customer retention

The best explanation of this concept I’ve read is from Henrik Kniberg in his post called Making Sense of MVP (Minimum Viable Product) — and why I prefer Earliest Testable/Usable/Lovable where the concept is summarized in this image:

Image from https://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp

The essence of agile software development is the ability to deliver value.

Value is something that a user can use in its entirety. It is not valuable for a user to log on without anything else working. It is not valuable for a user to pull a fancy report when the data is incorrect or incomplete. It is not valuable for a user to be able to log time without seeing the results. It is valuable only if it solves a problem.

So each iteration — or sprint — set a goal based on a problem that needs to be solved.

The essential product development process

The graph below attempts to summarise and visualise the essence of agile product (software and non-software) development following this spiral model, with each iteration or sprint following an evolving value-added process going through the same checkpoints on its journey.

Image copyright www.funficient.com

The four main checkpoints, or road signs asking you to stop and compare actual with reality, consists of the Deming P-D-C-A quality cycle, briefly explained below:

1. Plan

Planning is all about establishing goals. Translating this part of the process to Scrum would typically include backlog grooming, estimation and planning meetings.

What are you planning to do and what is the highest priority? What’s your MVP for the next iteration and how will you know you’re successful?

Translating it to the Lean Startup (for more on the difference between Scrum, Lean and the Lean Startup read here), this is defining a hypothesis to test.

What is the first experiment and how will you measure success?

Translating it to design thinking — which essentially is focused on the design of the process and mostly includes the aesthetics — it would be framing the problem — either in a problem statement or a How Might We question format .

The planning phase is also about assessing the risks associated with the solution.

What are your assumptions and what could possibly go wrong? What are the riskiest assumptions that needs to be validated first?

2. Do

Once you have a goal set - or thought about what needs to be done- the next step is to start building it- or the actual doing the work.

This includes all the activities needed to implement the plan, including changes to the plan where needed. It also includes testing to make sure that all the features work. It does not, however, include automated regression tests as it doesn’t make sense investing in regression tests when you are not yet sure that the product will be accepted by the customer. For more on my perspective on automation testing read here.

Translating this to Scrum it includes the daily standups, updating of the Kanban (or Scrum) board and everything usually handled within a single sprint.

3. Check

Once the team has developed something tangible, valuable and complete they show the working software or product to the user to validate. The main difference between user validation and internal test is that the user will have systems and processes that does not form part of the solution as part of their workflow. They will typically use the product in combination with other parts of their process — not in isolation — for it to be considered valuable.

The check-phase is intended to be an opportunity for feedback. If the customer doesn’t use it themselves, you can’t consider it checked. It is only checked once they’ve physically used it.

Translated to Scrum talk, this is the Review session. Translated to Lean Startup it is the testing of the hypothesis and translating it to design thinking it is the user testing.

4. Act

Feedback is as valuable as what you do with it. It’s of little use knowing you’re behind schedule but you’re not changing the plan. It’s of little use when a customer tells you something important in the delivered product doesn’t solve their problem and you put it on the bottom of the backlog because it wasn’t planned.

There’s really no point in showing the customer what you’ve done if you’re not planning to take their feedback as input into the next planning.

Feedback is as valuable as what you do with it.

Acting means that you are responding to changing requirements. It’s about spending time to gain insights that will make you (as a team) as well as the product (what the team is building) better.

And that is what agility is essentially about. A core learning loop with fast feedback.

It means that you’re building small pieces and giving the customer tangible testers to make sure that it’s what they really need and want. It means that sometimes the priorities change. Sometimes the entire design changes. Sometimes the whole iteration’s effort is thrown away (learn more about how to practice this skill with Improvement Kata’s) because it was the wrong thing. But it is exponentially better to realize you’re doing the wrong thing after a week or two than wait a few months to find out.

The act phase can be translated to the retrospective and part of the review. It is about going back to the drawing board and looking back at the past iteration actively looking for ways to do it better next round. Maybe you needed better data to test. Maybe you needed to get a test account authorised and setup before you could start testing. Maybe you needed a component library that you could use rather than build everything from scratch.

What can you next time to make it a little bit more efficient and smooth?

Translating this to the Lean Startup it would be the pivot or persevere question.

Do you keep going? Or do you change direction?

From a design thinking perspective it is iterating the design or coming up with new prototypes that better enables the user or solves the problem, knowing that with each iteration you understand the problem better.

Rinse. Repeat.

Once you’ve completed a full cycle, rinse and repeat, each time adding more value.

And that, my dear reader, is the essence of agility as I see it.

If you need help on your journey towards agility, visit www.funficient.com and book a complimentary call to discuss your problem and see how I can help.

Originally written on medium: https://medium.com/swlh/the-essence-of-agile-product-development-explained-601a056d8882