From Requirements to a Backlog
Quite often I get the opportunity to review an “Agile project” that isn’t working as well as it is supposed to me, and I start working with the teams to see how I can help.
A recurring issue in these scenario has been the lack of shared understanding about should be a basic Agile/Scrum concept – the backlog.
In many of the Agile projects – and yes there is considerable debate on what are the minimum Agile values / practices / principles that need to be in place before a project can be considered, but for now I’m referencing any project that is self-identified as Agile – the lack of a viable backlog is the biggest project risk.
And many of the Agile practices and principles are connected to risk management. Surfacing assumptions as we do in Agile projects is risk identification. Validated learning which is the root of both demos and retrospectives is active risk mitigation and so forth.
Let’s go back to the basics and establish what is and isn’t a backlog.
What a backlog isn’t
A wish list of everything and anything even tangentially related to the domain
A list of “requirements” generated years before that have been “converted” to user story format
Uncompleted, unverified fragments of “requirements” captured in multiple excel files
Fundamentally, a backlog is a holistic representation of what is valuable for an effort – the problem to be solved, an identified potential solution, plan to deliver that solution incrementally and iteratively as well as the team’s own learning curve.
More specifically, in terms of requirements a backlog should have:
User stories that have been written to support the project vision and to deliver the minimum viable product at least
User stories that are complete, i.e. with acceptance criteria and supporting artifacts for design and architecture so that development teams can begin work on day 1 of a sprint
User stories that have been prioritized by the product owner and estimated by development team and so on
Too often, Agile projects begins with “oh yes, we have a backlog so we start sprints now.” But very little attention is paid to the nature of that backlog and whether it is a good basis for project execution.
So how do we arrive at this scenario?
Let’s walk through some probabilities…
Most projects decide to go Agile when the development phase begins. Which means that usually the requirements and design phases have already been completed for this project. And the people responsible for the original requirements output were highly unlikely to be Agile practioners.
The requirements gathering process most probably focused on exhaustive coverage of everything anybody has ever considered, instead of targeted coverage of high-value functionality. The orientation would have been on “What” and maybe “How” to the exclusion of “Why”. The “What” and the “How” would have been based on the users’ knowledge and comfort level, usually the existing application.
Then when the decision is made to “go Agile”, the requirements document was converted to user story format and called a backlog.
This results in substantial sunk cost which is then used to weigh every subsequent requirements management decision. Since clients who have already paid significantly to have output generated don’t wait to pay someone else to “redo” the same process.
Given this background, how do you go from this so-called “backlog” to a true Agile backlog that can be the backbone for value delivery.
Mapping From Requirements to an Agile Backlog
Step 1. From Requirements to Features
Review specifications documentation (or any other available documentation) to identify category groupings
Start with standard functional categories for software projects, ex. reports, administration, data input, data analysis, etc.
Focus on identifying distinct processes (and/or set of processes that deliver shared business outcome) and grouping them as appropriate
Pragmatic recommendation is to print each requirement on separate piece of paper to make it easier to group requirements into categories
A general rule of thumb is aim for between 12-20 categories, depending upon the scope of your project.
Identify business value for each feature
What is the intended outcome – decision, output, knowledge – of this functionality from the end user perspective?
At this stage, this process might involve logical deductions by business analyst (maybe in conjunction with available subject matter experts, either in the team or within organization but not client at this stage)
Resulting output of the first step (MVP) is list of functional categories (features) with presumed business value; preferably, feature list plus more detailed business value identification, including mapping multiple processes for each feature.
Step 2. Reverse Engineering via Feature Mapping
Create Preliminary Feature Map by following the process described below:
Map features to indicate sequentiality or other preliminary dependencies such as integration with external applications.
Technical review of feature map (with architect or other tech lead) – preliminary architectural runway
Identify any new/different technologies that can be used to enhance this project
Identify any technical constraints or technological gaps from current infrastructure
Create just enough technical architecture documentation as needed (map of as-is architecture then annotated with identified gaps and proposed enhancements)
Customer review of feature map (facilitated workshop or JAD session format)
Walk-through feature map with product owners (customer/user group or representatives) to ensure that:
All essential user functional/business activities are reflected in the feature map
Any processes that do not add value are acknowledged and removed (note: may require customer acceptance to designate work as out-of-scope and removed from project requirements)
Roles/personas for each feature are identified
Have the product owner prioritize features based on current business value; reassure them priority can be adjusted as project evolves with no risk since they will be able to refine and re-prioritize backlog for each sprint and/or release.
Capture customer-identified gaps and dependencies in high-level functional design document. Include any flip chart drawings or other artifacts from the discussion
Generate a personas file and review with customer
Resulting output – Reviewed and accepted feature map with high-level technical and functional documentation
Obtain acknowledgement from customer that project success is now defined as delivery of value from feature map. Requires explicit acknowledgement by all that original requirements are no longer used/referenced
Before we go on to step 3, select the first feature to deconstruct
Best practice is to choose either the smallest feature for new teams or a feature that has average number of processes based on feature map completed above
Recommendation is not to start with most critical component – there is too much risk of not identifying key process gaps or dependencies
Another option is to sort feature list by technical complexity (by development team or tech lead), and have a separate feature list sorted by customer value. Generate weighted scale value to balance technical complexity and customer needs for balanced risk management.
Step 3. Feature-level Value Stream Mapping
Expand upon the select feature in this step to enable high-level understanding of the scope of functionality involved in delivering this feature. The idea here to go broad before we go deep; there will be considerable changes at the individual user story level as we work to deliver this feature functionality, but no new epics should emerge nor should there be any significant technology surprises.
Generate a step-by-step outline of the process user would take to complete each action, with one step on each index card or sticky note [follow from concept to cash principle]
Resulting structure would be something like a skeleton with the essential process along the “spine” and dependent processes dangling like arms and legs.
Keep in mind that a single feature story can have multiple processes, so you could end up with more than one skeleton for each feature.
Use different color notes for questions, gaps, dependencies, etc.
Review with customer or subject matter expect to set shared high-level understanding of scope for the feature. The business stakeholders will focus on the “why” and the “what” while the delivery team will determine the “how”, i.e. how best to deliver the “why and what”.
Step 4. Generate high-level user stories
Based on the results of feature map, generate high-level user stories or epics that will represent the current understanding of what work is needed to deliver this functionality.
Here are some tips on building the backlog of user stories from the feature map:
Each element of the process map should be a high-level story
Do not need standard user story format at this time. All you need is three or four words for each user story (verb + noun + phrase). Focus on the intent of the user story, not its format
Focus on dividing the process into user actions.
Start with user triggering process on-screen and then determine what needs to happen behind the scenes to deliver user request.
Finish with how the user is made aware of result and what, if anything, the user needs to do
Step 5. User Story mapping
The initial backlog of user stories will need to validated and expanded to represent the full scope of work needed. This step minimizes the amount of injected or discovered work and therefore will improve the forecasting reliability at a high leve
For each feature, rearrange user story level index cards or sticky notes into separate processes or for related functionality.
Complete user story level map for each feature – another “skeleton” output.
Identify business value for each story, i.e. how does this fit into the business process map.
Support business value user stories with technical enablers or other NFRs.
Review for clarity & dependencies.
Relatively size user stories – small, medium, large -- by using different color stickers on the cards or other low-effort approach. Anything bigger than a large is cue to split into two or more user stories.
Now you're ready to start writing user stories for the highest priority items.
Step 6. Release and sprint planning
Determine the size / cadence of releases and sprints
When possible, plan releases a high-level by association features with releases
Decide value for first release, whether it is primarily one feature or a combination of no more than 3 features
Define release backlog (at least for first release) by using feature and user story maps
Step 7. Validation and verification
At the end of the first sprint, validate the story map. Note any gaps or other work identified during the course of the sprint. Update release backlog as needed. At the end of every sprint, always ensure you’re reviewing/validating release backlog and feature maps.
Repeat steps 3 and 4 for each additional feature.
Keeping true to the Agile value of just enough and just in time, do not deconstruct all features at once. Best practice is to deconstruct one feature all the way through the above process and then deconstruct another feature or two through step 3 and resume with step 4 for those features after first feature has been completed.
I’ve applied this process at several complex organizations, both in governmental and commercial environments, to achieve substantial improvement in the requirements process especially in terms of greater business value through a more realistic understanding of scope and improved prioritization and estimation as well as the removal of not-needed or not-needed-enough work.