Product Backlog Grooming

What’s exactly the product backlog grooming in Scrum

When reading most of the suggested books about scrum, the grooming technique is often cited, but it is not officially normed as a scrum meeting or technique.

This lack of officialization, in my opinion, is an important cause of uneffectiveness in product backlog management and adjustment and, thus, impacts on the team’s performance.

I searched for the word ‘grooming‘ in the official scrum guide and these are the only two statements found:

“Product Backlog grooming is the act of adding detail, estimates, and order to items in the Product Backlog. This is an ongoing process in which the Product Owner and the Development Team collaborate on the details of Product Backlog items. During Product Backlog grooming, items are reviewed and revised.

However, they can be updated at any time by the Product Owner or at the Product Owner’s discretion. Grooming is a part-time activity during a Sprint between the Product Owner and the Development Team. Often the Development Team has the domain knowledge to perform grooming itself. How and when grooming is done is decided by the Scrum Team. Grooming usually consumes no more than 10% of the capacity of the Development Team.”

The technique allows the ProductOwner (PO) to effectively implement what, in the traditional project management discipline, is called as rolling wave planning techique: decomposing and adjusting the scope of the product.

It has been thought in Scrum, to help the ProductOwner in:

• decomposing the product backlog features and epics, in more manageable new user stories
• preparing the stories for the next sprint
• have the team estimating any new story (comprising the ones resulting from the decomposition just mentioned)
• have the team re-estimating any existing user stories
• taking under-control the product backlog and its compexity to effectively plan for the releases

According to what above reported, the grooming is a key meeting that I don’t want the team forget and, hence, I suggest to insert it right in the middle of each sprint and the team can dedicate 1/4 hours.

One of the usual thing that will happen during the grooming, is that the ProductOwner asks the team to decompose an epic into new user stories that must be estimated using the planning poker game.

Let’s say for example that we have a story like the one below and its estimation is 21 story points:

As a student of the college portal
I want to register my account, search for and enroll to an exam
so that I can save time

The ProductOwner knows that such story will not fit a sprint. During the grooming the PO asks the team to help her to decompose it.

The story, for example, is decomposed in three different stories:

As a student of the college portal I
want to register my account
so that the system remember my credentials

As a student of the college portal
I want to search for an exam
so that I can plan the  necessary time to study and prepare the exam

As a student of the college portal
I want to enroll to an exam
so that I can save time

Each story, finally, must be estimated and the team proceed playing the planning poker game.

Agile Raw Affinity Estimation

Let’s say You have just finished to create the user stories for your product and, a bit proudly, your product backlog now counts two or three hundred of items. How can you estimate them, effectively?

It is not always possibile to have all the items of the backlog estimated using the planning poker game. This estimation is somehow precise, but it is actually time consuming and not practicable in one-shot against the whole backlog.

Having a quick first rough estimation of the whole backlog, however, could be the right answer in order to start planning for the development of the product, dividing the effort in releases and giving some forecasts to the management, in terms of schedule and budget.

Actually, we are assuming that these first estimations won’t be perfect but, at last, what estimate is perfect? And moreover during the initial phase?

A technique that could help here, is the Agile Raw Affininity Estimation. I usually organize a workshop of 3/4 hours, within which the team usually can estimate more than two hundred stories.

Participants, material & preconditions

The participants involved are: the ProductOwner, the ScrumMaster and the core team. One important constraint is a logistic one: the room where you are going to hold the meeting, must have wide walls where to stick the user stories.

This is the material you will need:

• A4/Letter sheets
• Markers
• Post-it notes
• Paper tape

And these are some pre-conditions:

• well-formed stories (U-INVEST),
• the ProductOwner should have already presented the stories to the team even if at high-level (epics)
• the stories must be all printed each on a A4/Letter sheet (take care of the font size you use: every story must be clearly readable from 2/3 meters away from the wall) ,
• a printed copy of the team’s definition of done.

These are the main steps of the technique:

• Workshop explanation
• Silent estimation
• Story points estimation
• Estimation review
• Consolidation

Workshop explanation (Coach/ScrumMaster)

The Coach/ScrumMaster explains the team and the ProductOwner, the scope of the meeting and come into a detailed explanation of the process’s steps.

It’s important that s/he stresses the rules underlying the agile estimation (story points, complexity, relative weight, delphi, etc.) and remembers the team that each story estimation should aim to satisfy the definition of done the team already decided.

The Coach, then, sticks on the very bottom-left side of the wall, a sheet of paper reporting the word “SMALL” on it and on the opposite bottom-right of the wall another sheet with the word “BIG”. Finally, he rolls the tape from left to right, joining the two words.

This because, the team will hang the stories on the wall according to their size, forgetting, for this first step, about the estimation in figures, but reasonging about a gut estimation and positioning them in respect of the scale (SMALL>BIG) and of the other stories already attached (relative estimation).

The ProductOwner, then, gives the team the pile of user stories already prioritized (top of the pile most important stories).

Silent estimation (Team)

Now it’s time for the team to step in. During this phase the team remains silent. Every member has two possibilities: estimate a new story or change the estimation someone already did of a story.

Estimate a new story

In turn every team member, take a story from the pile, a piece of paper tape and stick the story to the wall, according to his/her perception of the story’s complexity (SMALL vs BIG) and in respect of the other user stories already hanging.

Change an existing estimation

The team member do not take any new story from the pile.

S/he detaches the story to re-estimate and sticks it again in a new position, according to her/his complexity perception and the other stories already hanging.

The process above described, continues until all the stories have been estimated.

Every team member can ask clarification about the user stories to the ProductOwner.

Story points estimation (Team)

Now it’s time to transform a position on the wall, in the corresponding estimation in story points. My suggestion here is to use the Fibonacci series.

The team writes each number of the series on a single post-it (1,2,3,5,8,13,etc) and attaches them near the line of the scale.

 Estimation review (ProductOwner)

Well, the team now deserves a break and a coffee and leaves the room.

The ProductOwner, which remained silent during the whole estimation process, could necessitate some clarifications.

Thus, she takes small orange/red post-its and attach them on each story that needs a clarification about the estimation done.

When the team comes back, it gives the clarifications the ProductOwner needs about the estimations. Sometimes it happens that after the discussion, the team has new elements and decides to change the estimation, moving the stories down the scale.

Consolidation (Team + ProductOwner)

This is the last step of the workshop.

The team and the ProductOwner write on each user story, the right estimation according to the position and the corresponding number.

The ProductOwner, finally,  gathers the stories and inserts the figures into hte product backlog and that’s it!

Who is still saying we agilists do not plan at all?!

One of the worst myths of agile, is that people think agile means to not plan at all. Actually, we do not plan in a traditional way, beacuse we use the rolling wave planning technique and we make a great use of visual management.

When starting a new agile project, there are some mandatory plan steps that, in my opinion, every organization should take. Before the first line of code is written, management needs estimations, even raw, dates, plans; this to understand how to finance the project.

Usually, we have a bunch of events we should keep into account.

First of all, a first high-level requirement gathering must be done in order to have clear understanding on what main features the product should have, according to the business value to reach.

This is the responsibility of the ProductOwner, who is the client or a representative of it.

Then, the product roadmap must be arranged: start and end date of the project, release dates, any important deliveries, final UAT, main milestones in terms of operations, marketing, etc.

Yet, now it’s time to arrange the user story mapping workshop (see here), to start creating the first version of the product backlog.

Ok, ok, I know. You already did a lot of work and you want to start asap with the first sprint. But, please, listen to me, calm down.

There are some last steps to perform.

To start the next sprint, you need to be sure that the most important user stories, the ones the team will work on immediately, are correctly decomposed and fine-grained. So, you should arrange a meeting with the ProductOwner and the team (Backlog Grooming), to be sure the stories are sized properly according to the sprint lenght and the team velocity.

Ok. Now it’s important to estimate the complexity in user story points, of each story and, indirectly,of the whole backlog. This will help you to make the release and iteration planning.

Use the planning poker game it’s not feasible when estimating too many user stories: it’s too much time consuming. One of the technique I use is the raw affinity estimation, which I’m going to describe shortly in one of my next posts. This technique let you to assign a first rough estimation to all the product backlog stories.

Now, you should have enough data to plan the product deadlines in terms of product release.

You [should] have:

• the total complexity of the product backlog,
• the velocity of your team/s,
• the main milestones

This leds you to the release planning (see here) where you put the epics (main features) on the timeline according to data above.

Finally, you are ready to sprint!

A lean approach to portfolio definition

When adopting agile as a product development methodology, the organizations wouldn’t forget to manage their portfolios accordingly.

It happens frequently that when organizations want to adopt agile, they think it’s only a matter of introducing just one more new methodology in the “assembly line”. They wrongly imagine that it’s only a matter of having the teams doing new things or, either, performing the old ones, better and more efficiently.

What they usually lose, is that introducing agile means to be ready to change organizational habits, behaviours, rules, policies, processes, tools, etc.

The portfolio management is one of the practices that should change.

Ok, let’s imagine you must arrange a portfolio of three different but inter-connected products, which will be developed using agile methodologies. You should have a first raw list of the main features each product should have (EPICS).

Then, You should arrange a workshop in which the most important stakeholders should attend: product managers, program & project managers, CEO, etc.

Be sure you will have post-its of different dimensions and paper tapes available and a large wall where to stick them: you will make great use of some visual management techniques.

Start the meeting, deciding the time-frame within which the products should be deployed.

Then, write the name of the products involved on the Y axis.

Now it’s time to ask the first product manager (or ProductOwner in SCRUM) to write each feature of the product on one post-it and hang it to the wall, positioning it on the timeline where it is supposed the feature must be available. This first chronological prioritization, must be done according to the business value each feature is supposed to bring to the organization.

Then it’s the turn of the second product manager.

Finally, the third product manager does the same.

Now it’s time to analyze and verify the interconnections between the products, asking the three product managers and the other stakeholders, to recognize and highlight any dependencies. This is one of the most important moment: the dependencies, indeed, can be conditioned by several factors: technical, tactical, financial or business ones; thus, choosing the right attendees for this meeting is paramount. It happens that during this phase new features arise.

According to the dependencies just drawn, the attendees must reason on how to reconcile every concern and necessity, by moving the features anticipating or delaying their availability.

Once the reconciliation above is finished, it’s time to analyze risks and impediments. This is a brainstorming session: each person writes on a post-it a risk or an impediment and stick it apart from the portfolio chart just defined.

The facilitator, arranges the items by category, clustering them, remove the duplicates and when the list is consolidated, asks the attendees to sort them by importance. Then, s/he assignes an identification number to each.

The facilitator, then, asks the audience about the impact of each impediment against the portfolio in terms of features, dependencies  or even months and/or products.

The workshop is almost finished. Now it’s time to collect the features and the related attributes (sequence, dependencies, dates, impediment references), into the product backlogs and the impediments list.

More reading here (see the book list page):

• Scaling Lean & Agile Development – Thinking and Organizational Tools for Large-Scale Scrum Craig Larman – Bas Vodde
• Scaling Software Agility: best practices for large enterprises, Dean Leffingwell
• Lean-Agile Software Development: Achieving Enterpise Agility Alan Shalloway, James R. Trott, Guy Beaver Net Objective Lean Agile series

Be good

User Story Mapping

Let’s say you are ready to start sprinting using SCRUM: have you already created a first raw version of the product backlog? Well, why not using the story mapping technique?!

When a ProductOwner find herself, for the first time, involved in the product backlog creation, it’s usually a real drama. This because she has to take into account many things contemporarily: the high level product’s vision, what the product should actually do as macro-functions, what are the relative micro-functionalities, identify any dependecies among them, etc.

The ProductOwner (PO) should work on these activities with the whole team, in order to get it completely involved and to let the team start visualizing the product, its functionalities and think pragmatically about it.

An interesting technique I usually use when coaching teams, is the user story mapping workshop, which  can last 4 to 8 hours, depending on the product complexity.

The first thing I like to propose, is to ask the PO to give a brief description of the product and its main functionalities and characteristics.

Then, the ProductOwner, writes down on post-its, the main activities a user can do with the product and hang them on the wall following the horizontal line of the timeline, sequencing them chronologically and the by priority.

For example, if the product were an email client, we could have as main activities: manage emails, manage calendar, manage contacts, etc.

Now it’s time for the PO and the rest of the team to identify for each activity, the relative tasks. In our example, the tasks identified for the Manage Email activity could be: Compose email, Read email, Delete email, etc.

Great, well done! We can proceed, now, defining the actual user stories that, as you know, must satisfy the INVEST acronym:

Indipendent (preferably not coupled with other stories)

Negotiable (the way it will be implemented can be negotiated)

Valueable (the story must be valuable for the ProductOwner)

Estimable (the story must be estimable by the team)

Sized-properly (the story must be small enough to fit, with other stories, into a sprint/iteration)

Testable (the story must be testable)

Coming back to the previous example, the task Compose email, could be decomposed in the following stories: Create basic email, Send RTF email, Send HTML email, Set email property, etc. Keep into account that, as Mike Cohn teaches us, a story should have the format “As a (role) I want (something) so that (benefit)”, the stories above reported for the sake of brevity, are not represented that way

This is the final step of the user stories creation, now the ProductOwner can gather the stories and write them down into the Product backlog. Most of the time, this first workshop won’t produce fine-grained stories, but it’s a mandatory step that will be followed by a new workshop usually called ‘Backlog Grooming’, where the team and the PO, work togheter on each story, in order to decompose it and analyze any dependencies.

Finally, before you can start with the first sprint, you will have to estimate the entire product backlog. It happens usually that the stories contained into the backlog, are hundreds; estimating all them using the planning poker game, is not feasible because it’s too time consuming. Actually, there’s another effective technique I’m going to talk about the next time, which explains how to estimate hundreds of stories in few hours, but please be patient for that.

And this is the result of one of the workshop I facilitated

IPMA Portfolio Visible Planning® for an Agile Implementation

Saturday November the 10th, I’m helding with Luca Cavone, an agile workshop about about lean portfolio management for an agile implementation.

Click here for additional information.

Scrum metrics: when to use?

Metrics are numbers, figures; they cannot represent reality….but they could be triggers that help in taking decisions.

Metrics, in software development, are usually requested by managers that want to know about the teams’ performance.

Agile teams usually know very well about their performance and the quality they are putting in. They also know quite well what’s the magic formula, that is different for every team, to balance  effectively perfomance and quality.

One day it happens, though, that pressure starts to grow.
The ProductOwner asks for some more functionalities, usually because of some changes about new ‘absurd’ deadlines. Obviously, the team tries to find different solutions or compromises engaging both the ProductOwner and the management people, but business is business they say.

The team struggles in balancing the requested increased level of performance, with the high quality standards agile prescribes, whatever XP, Scrum, Kanban or any other approach you are using. The healthy practices like pair programming, test driven development, refactoring, code inspection are slowly but progressively put apart.
This situation is disruptive for quality as well as for the team.

But, what I’ve noticed is that providing the right metrics to the management, could be the key factor that helps teams to face the situations like the one just mentioned.
Let’s see how.

First: your metrics obviously will report performance data like the velocity (user story points finished within each iteration).

Then, remember to associate to the velocity, also the metric ‘remaining story points’ (story points related to stories not finished within the sprint); that’s a metric that can mean at least two major things:

1. The team wronlgy selected too much stories (e.g. due to estimation errors)
2. The team was forced to select (consciously or unconsciously) too much stories (e.g. the CEO decide a new deadline and the PO pushes for more stories)

Usually, if the problem is how the team makes the estimations, this number will decrease progressively the next iterations, until it will reach zero.
When, on the other side, the PO and/or the managers push for releasing more stories, that number won’t decrease. This situation is also witnessed by other qualitative indicators.

Gather other qualitative information like the coverage of the unit tests, the # of test successfull/failed, # of issues closed vs new ones, # of broken builds, can help to study the qualitative trend of the product.

This data should be automatically collected by continuous building and integration systems and, again, automatically aggregated in database, in order to let the team remain completely focused on the development.

Once I have that data, I usually create two different graphs: the radar and the line graph.

The radar graph represents the results of each sprint, joining together different indexes. This kind of graph, actually, gives also a visual representation of the covered area: the larger, the better. The line graph helps to see what’s the underway trend of quality, verifying for example if some corrective actions done in the past, have brought any results.

Hence, let’s first concentrate on the radar graph. What I make is to transform the data just collected (see above), in qualitative indexes (e.g. from 1 to 5: 1 poor…5 excellent). Then, I calculate a Quality Index (QI), which is simply the sum of each index.

I’m not yet satisfied.

I do not want to assign the same weights to all indexes. Indexes like the # of tests sucessfull and  the coverage of the unit test or, furthermore, the # of broken build, must have different weights.

The successfulness of tests are something that I assume is somehow taken for granted. When a developer writes a unit test, I assume that she writes the code correctly.
Talking, instead, of indexes such as the broken builds or the coverage, I consider them much more important, because break a build or have a low unit test coverage, is something really bad.

This is the reason why I assign weights to the metrics I’m studying, like the ones below reported.

Now it’s time to recalculate the indexes according to the weights just assigned to each.

And finally, my graphs magically appear!

Here it is the radar graph where every axes represent an index.

The line trend graph is displying the QI, that is the sum of all the weighted indexes.

Remember, metrics are numbers.

Do not rely only on them when it’s time to take decisions: use this data to prove and certificate your feelings, impressions.

 Often your gut feeling is better than any metric!

Have fun

What to do with simple, complicated and complex systems?

Can you recognize systems and what approach is the best one for each?

Ralph Stacey with his ‘Stacey Matrix’, helped us in recognizing the main system typologies, what characteristics they have and how to cope with them. This matrix is quite simple.

The X-axis of the matrix represents the level of certainty on how the system works. More specifically, what are the relationships and the linkages between the cause and effect: in few words the behaviors of the systems in response to predefined stimulus.

The Y-axis represents the level of agreement on the system’s behavior just mentioned, among us: group of people or project team.

When a system has a finite set of responses, which can be easily related to an input, it means that the relation between cause and effect are clear. These systems are called simple systems, which can be approached with the best practices, taking advantage of previous experience.
If you are wondering you should use agile with projects related to those systems, I suggest NO. It is not necessary.

If we only move the slider along the X or Y axis, diminishing the level of certainty or agreement, we fall into the area of the complicated systems. These system are less predictable. They still continue to be understandable, but not in advance and only after a deep inspection, looking at its internal mechanisms and rules. With systems like these ones, we can apply good practices, which help you to get closer to the solution.

Let me give you an example: think for a moment of a watch.

If you aren’t a watchmaker, a watch is something, at least for me, very, very complicated. But I’m certain that if we spend time analyzing its gears, wheels, screws and how they are related each other, sooner or later, we will be able to disassemble and reassemble it, let’s say, “quite easily”.
If you are wondering you should use agile with projects related to those systems, I respond SOMETIMES IT COULD HELP.

Moving far from certainty and agreement, it arrives a place that is called “complexity land“.

Within this land cause and effect cannot be nor predicted, neither analyzed, but only perceived in retrospective.
In such a land we must be awake, attentive and plan for few steps at a time.

What is most important, then, is to analyze the results of any action, in order to decide whether to continue that way or change our plans accordingly to the feedback obtained. If you are wondering you should use agile with projects related to those systems, my answer is YES, YOU MUST.

The successful pattern here is:

1. break complexity up in small chunks,
2. plan iteratively for each chunk,
3. act,
4. inspect the results,
5. adapt your future plans accordingly to the feedback just obtained.

Finally it arrives the realm of the chaos.
A realm where certainty and agreement are foreigners and the only approach here is to explore such a realm.  The only suggestion I can give you here, is good luck!

Why automate testing?

Are you really sure that only putting testers in the same room as others team members, let you deliver free-bugs & high-quality software?

In the previous post, I wrote about how to bring testers in the war room, with the other team members, identifying a possible agile testing process.

What I tried to describe was a raw hypothesis of what testers should do to cope with an agile enviroment: collaborating with developers and ProductOwner, writing test scenarios and test cases, writing automation tests and executing them, as well as performing exploratory and performance testing.

The scope of this post, actually, is to convince the ones of you not yet convinced, that it’s time to start automating tests, by telling a “nice” story.

Firstly, I would suggest the book “Agile Testing: a practical guide for testers and agile teams” written by Lisa Crispin and Janet Gregory. Several of the ideas and suggestions I gathered about agile testing, actually come from such a book.

Why Automate?

When developing a software, the time inexorably passes and, the application being developed, becomes bigger and bigger, and complexity increases; it will soon take too long to have it completely manually tested.

A condition like that, is primarily responsible of three common drawbacks:

1. team velocity decreases because developers start to help testers in their testing activities, instead of develop
2. or/and costs increases, because more testers are put into the team
3. or/and quality decreases because of the user stories start to be delivered not fully tested, due to few time and too much things to do

A real nightmare, isn’t it?!

What usually happens in a scenario like that, actually, is that testers start to struggle because of the few time available. They inform of such a situation the whole team during the daily meeting.

The ScrumMaster, in order to promote team self-organization, suggests that developers could help testers in executing manually testing, to solve that temporary problem. This choice drives the whole team to deliver less stories, respect to what initially committed, and a sensible decrease of the velocity starts to be detected.

The ProductOwner is daily informed about the situation, he starts to be worried.

During the next retrospective the team reflects about the problem, concludes that is not a temporary situation, hence decides to ask for new testers.

New testers arrive and it happens that, according to the brooks’s law, it takes more time that estimated to have them completely productive because they have to learn and practice and often pairing with others testers. At the next demo the team presents again even less stories respect to what was decided. The ProductOwner continues to be worried.

The ProductOwner then starts to push the team in having more stories released, asking and obtaining to avoid some quality checks and having less testing.

Software quality starts to decrease and this trend is showed by looking at the quality data gathered at the end of each sprint: Unit Test Coverage, Test passed/failed, broken builds, software quality metrics.

The technical debt increases, daily, and the ProductOwner decides to bring the problem to the management, asking for a refactoring sprint, that actually further delays the final delivery.

What about start automating tests?

There are many tools available and most of them are open  source tools, which cover all of the following categories:

• Testing WebServices
• Testing Web Application
• UAT and Integration Testing
• Data Generation Tools
• Database testing tools
• Performance Testing Tools
• Database Usage
• Application Bottlenecks and Memory Leaks
• Security Tools