About Story Points, Estimations and other expensive misconceptions

TL;DR
Have you ever wondered why navigation software like Google Maps is so accurate when it comes to predicting expected arrival time when planning a route?

The answer is relatively simple: Google Maps continuously combines static data (such as distances) with dynamic real-time data (traffic jams, train delays, weather-related closures, and more) and continuously adjusts the estimated time of arrival as you travel, based on constantly changing environmental variables.

If we swap Google Maps for any project management tool, distance for project scope, and traffic conditions for sick days and interruptions due to other projects or bugs, we inevitably have to ask ourselves why we still try our luck with software projects using such static methods as planning months-long roadmaps, multi-week sprints, and time estimates months out.

Where other industries are already much further ahead, the driver of digitization of all things is sticking to methods that are based on hair-raising assumptions, ignore known uncertainties and gloss over far-reaching dependencies. The result is regularly horrendous project costs, desperate employees and annoyed customers, as well as many preconceived opinions about who is to blame for this mess.

But how do we move from almost static planning to dynamic forecasting that adapts to changing environmental conditions and can provide meaningful information to all parties involved?

The ritual

Let’s first take a look at a typical process in software development.

It’s almost a ritual that repeats itself every few weeks: user stories are created, the team meets in meetings, the question of “how long?” comes up. It is answered either in the most classic currency, time. Or in the somewhat more progressive variant in Story Points. They are an abstraction of time, often referred to as “complexity”, and are often created on the basis of mathematical number sequences (the Fibonacci sequence, for example, enjoys great popularity).

At the end of the meeting marathon, there is a backlog with all the user stories that the team could agree on (or that were resignedly accepted at some point). This backlog serves as the basis for the next sprint, a two-week or longer time frame that has the complete processing of the backlog as its stated goal.

The frequent result at the end of the sprint: the goal has not been reached, there are still stories left, more have been added to the sprint in the meantime and the bugs have not solved themselves. True to the agile procedure, the reasons are then discussed in a retrospective, where often only symptoms are addressed. In the end, depending on the perspective, the software developers are to blame or the product managers who do not have “their people” under control.

The fundamental question about looking to the future

You don’t have to be a scientist to recognize a deep-seated skepticism among many IT professionals about the concept of Agile frameworks, story points, sprints and estimates. It is a skepticism that is unfortunately rarely consistently questioned in the day-to-day work of a team. But what exactly is the problem?

In the beginning, there is the basically understandable desire to know when, with what effort and at what cost a new feature will be available. The problem is that a static figure (the cost) is negotiated on the basis of vague, individually different objectives at a time when important information is not yet known. This figure is interpreted as final and used for further steps (offers at a fixed price, follow-up planning for further projects, etc.). However, software systems are rarely off-the-shelf products, but often individual solutions with a multitude of dependencies, imponderables and unknowns.

The concept of sprints tries to counter this problem with shorter time intervals of a few weeks. Unfortunately, experience shows that even this is usually not enough. The software development process is too complex, too many players are often involved at the same time, and the effort required to operate existing software systems is often too great.

Ultimately, we try with this method to statically fix a complex process taking place in the future with many open questions and variables. It is not without reason that the arrival time of Google Maps route planning changes permanently while you are on the road and it is not without reason that weather forecasts are adjusted regularly. Anyone who has ever tried to plan the way home from work in advance to the minute and got it spectacularly wrong due to red traffic lights, delayed trains and buses, and a spontaneous detour to the supermarket should come up with a good answer to the question of why this static principle should work in a disproportionately more complex environment with many people involved and possible dependencies on third parties.

The devil is in the details

A systematic error is already evident in the planning of the sprints, a phase often referred to as backlog planning. Let’s take a new feature for an e-commerce store as an example to illustrate this: customers should be able to mark products as favorites, which should then be stored in a digital wish list in the store.

This feature is obviously decidedly too big for a user story, especially if we want to take a modern approach with CI/CD-based development and frequently deploy small and incremental changes. An initial breakdown into smaller bites could look like this:

• User Story 1: Each PDP (Product Detail Page) should have a button to mark the product as a favorite.
• User Story 2: The marked products should be able to be displayed via a “Wish List” link.
• User Story 3: It should be possible to remove individual favorites from the wish list page.
• User Story 4: On the wish list page, it should be possible to add individual products to the shopping cart immediately.
• User Story 5: On the wish list page, it should be possible to add all products to the shopping cart at once.
• User Story 6: The entire wish list should be able to be completely emptied via a button.

At first glance, we have here a complete list of features for a working wish list. The individual stories are manageable in size and clear at first glance, now we just need the software engineers’ assessment of the time required.

As developers, the most we can do is estimate new functionality based on experience we’ve already had. Developer Tom thinks that User Story 1 can be solved within one day. Developer Kirsten is not sure if the wishlist should only be available to logged in users or to everyone. The clarification of this question would at least require the presence of a product designer who, however, would not be able to answer this fundamental question without a certain amount of user research. Under certain circumstances, the business perspective also plays a role, which is often not represented as a role at all in sprint planning meetings. However, the answer to this question determines which technical steps are necessary for the implementation – accordingly, the developers cannot give a satisfactory answer, but can at best guess. Too many questions are still open at this point, such as:

• How are variable products handled (i.e., products with additional attributes such as size or color)? Are these attributes displayed directly in the wish list or does the user get to the PDP?
• What happens if a product is not available?

Often, attempts are made to clarify these questions during planning meetings. However, this leads to a very superficial view based on pure assumptions, which reduces the role of a product designer in terms of user research to absurdity. After all, do we know if the wish list would be used at all if people had to register and/or log in first? And if not, is it even worth implementing this feature? If such questions are decided in a planning meeting under time pressure, there is a great danger that the user will not be able to benefit from the development, which in turn will result in high costs for the company.

Experience shows that the process described in the last paragraph alone can quickly drag on for half an hour or more, and at this point we have only just talked about a single user story. This means that there is still no well-founded estimate, and with six user stories, each of which requires 30 minutes of discussion, the planning meeting alone takes three hours. At the same time, six user stories for a two-week sprint backlog is a very modest amount in most teams.

Back to Waterfall?

Now, one could say that the designers would have to conduct research in advance and present a finished design. Here, every professional product designer will flap his ears and reply that the input of the developers is necessary for such a process. A designer can hardly assess whether the technical effort required for the various possible variants is not already far too great from the outset. And what happens to a design prototype made long in advance (or in written form somewhat old-fashionedly often called a specification) in the short-lived world of software development should be obvious to anyone who has ever heard the term waterfall model.

So we are going around in circles. If we simply pass over the open issues at this point, we put developers in the extremely uncomfortable position of having to provide an estimate based on non-existent or missing/incorrect information. Here, every developer will understandably give a very conservative estimate, knowing the many questions that are still unclear. If we multiply this approach by many user stories, we end up with a number (e.g. 60 story points), but we are basically lying to ourselves and get the receipt as the duration of the sprint increases and the discrepancy between the predefined goal and the reality that does not match it becomes more and more obvious. This is at the very least frustrating for everyone involved and ultimately expensive for any company, and therefore not a solution.

Use story points instead of time?

So should we rather use Story Points instead of time as a unit for estimation?

When it comes to Story Points, we must be crystal clear about the fact that they are nothing more than an additional abstraction layer for time. According to their definition, Story Points should not (only) include time, but also the complexity of a user story. But (as we have seen above) the complexity cannot yet be captured at the time of estimation.

The estimation therefore becomes a pure guessing game. At the same time, we artificially increase the complexity by using story points. We let ourselves be misled by the idea that we have planned everything well (because we now have a static number in the form of Story Points), but in the end we know only frighteningly little about the concrete challenges that will await us during the Sprint.

In times when almost every industry is focused on maximum efficiency, this approach is still widespread, and one does wonder how serious management really is about the desire for efficiency and increased profitability when the only answer is even more dogged estimates and an even more vehemently expressed desire for a static planning figure.

But how do efficient, autonomous teams actually work and successfully overcome these challenges? How do we arrive at a maximally realistic forecast of the effort involved in developing new features?

About the art of omission

The first step is the easiest and, in terms of “it’s always been this way” behavior, also the most difficult: we simply forget about time estimates and especially story points and throw them out of our toolbox. I would like to explicitly exclude rough time estimates for new projects – we can certainly say in our example above that we probably shouldn’t need more than two to four weeks to implement the wish list in a typical product team with product manager, product designer and four to six software engineers. If you want to go a step further, you can even question the usefulness of user stories – a valid argument that I will address later.

Let’s first take a look at what we know at the moment:

• We know the approximate scope of the wish list feature.
• We know the existing system and can estimate where and how this new functionality will be implemented.

And if we’re honest, we don’t really know much more. At this point at the latest, the intensive collaboration between the roles involved in the implementation now begins: Product Manager, Product Designer and Software Engineers. The basis for an initial workshop can be a Low Fidelity Prototype, which (at the level of a pencil sketch) depicts a possible user flow as well as the basic user interface. This visual foundation is an important basis for all people to build a common understanding of the functionality. Questions will arise, ambiguities will need to be clarified, and technical feasibility will need to be explored. Depending on the complexity, a technical proof-of-concept may be necessary. This iterative process takes a few days, requires further steps such as user research, and must be concluded with a shared concrete understanding of an MVP (Minimal Viable Product). An MVP is the smallest possible functional piece of software that offers initial added value for the user, can be implemented in the shortest possible time, and is then further developed based on user feedback or discarded if the desired effect does not materialize. In this case, too, the company benefits from the fact that only the smallest possible amount of development resources was put into the implementation and the costs were kept as low as possible.

Small steps for a big goal

We now break down the scope of this MVP into small user stories that can be implemented as independently as possible. Each individual story should be small enough that the developers are confident that they can implement and complete it within one to two working days on average (whereby the Definition of Done explicitly includes automated testing and release). This scale is fundamentally important and a key component of dynamic planning. The underlying principle is simple: we can say with relative accuracy and certainty how long it will take us to get from our home to the bakery around the corner, but if we want to get to a place 1000 km away, it becomes much more difficult and imprecise. This is exactly the same principle we use for the size of the individual user stories: we cut individual functionalities into such small pieces that we can already discover and consider as many unknowns as possible during this process and thus maximize the confidence for an implementation without unpleasant surprises.

The result is a variable set of, say, 25 stories for our wish list. Variable because:

1. …we probably haven’t captured 100% of the MVP scope, but maybe only 90%.
2. …nothing is as constant as impermanence and we always have to reckon with unpredictable changes and events in our daily business.

25 user stories, each of which should not take longer than two working days at the most. Now, of course, we could start to calculate: 25 stories, each requiring two days of effort, six developers, three of whom work 80%, one of whom has two weeks of vacation, 30% of the time has to be spent on maintenance, etc. But that would take us back to the old way of doing things. But this would mean that we would fall back into the old pattern and try to define a static number. However, this will not work (see point 2 in the previous paragraph).

The Burn-up Chart

A burn-up chart, in contrast to the widespread burn-down chart, is exactly the opposite: instead of working towards the target 0 via a descending curve per sprint, our curve rises steadily. The gradually emerging intersection with the scope is the target here, but unlike the burn-down chart, it is dynamic and based on data points reflecting reality.

We start by implementing our wish list. Work with a burn-up chart and one-week cycles. Fully implement the first user stories and measure the time required.

• Story #1: 1 day effort
• Story #2: 2 days effort
• Story #3: 2 days effort
• …

Here we enter the sum of the number of User Stories implemented in the current week as a data point in our burn-up chart.

• Week 1: 3 User Stories
• Week 2: 6 User Stories
• Week 3: 5 User Stories
• …

With this burn-up chart, we can now regularly forecast when we will reach our full scope of 25 user stories (based on the most current data). We can make an optimistic assumption, a linear assumption, and a pessimistic assumption. We see that our “arrival time” changes, and it changes daily or weekly. We also see that new user stories that are added push back the time of completion, as well as other unexpected interruptions in team productivity. On the other hand, if we find that we could do without a functionality after all, the deadline shifts forward again.

So we can play with these values, incorporate environmental variables, and read a dynamic trend. Above all, we work with real-time data, knowing and accepting that it can (sustainably) influence the trend. We do not fixate on a static number, but anticipate in our work methodology the constant changes within a software project.

We no longer try to predict the future, but rather give a forecast, like a weather report, which also incorporates new information into its weather forecast on a daily, even hourly basis. But how does this method satisfy the need for an indicator of effort that is known in advance?

A dynamic environment cannot be represented statically

The honest answer is: we can’t. No one can, because no one can look into the future. All parties involved (employees, customers, management, board of directors, investors) have to learn and understand that the previous concept of static planning does not help anyone in the end, but only leads to (sometimes massive) additional costs. These come about through:

• Too much time invested in planning, which only a short time later has become obsolete again due to changing environmental conditions. This can be partners on whom one is dependent and who are faced with exactly the same problem. It can be employees who drop out or leave the company. It could be unexpected technical difficulties during ongoing operations. No company is immune to this and we all know that these are not isolated cases but the rule.
• Employees leave the company because they are not willing to accept the unnecessary pressure that comes with it. Many employees understand quite well whether what they are doing makes sense. But if they then have to take the fall when deadlines cannot be met, even though the underlying system cannot function at all, no one has to be surprised about high rates of departures.
• Crisis communication to partners that repeatedly conveys the message “Unfortunately, it will take longer” can be expensive. In addition to reputation, follow-up collaborations may be jeopardized and the company is quickly perceived as unreliable (although the partner company is probably no different).
• In poorly drafted contracts, contractual penalties sometimes include exceeding a delivery date. This is where a vicious cycle begins, with more pressure on teams and thus more frustrated employees. See point 2 – employees no longer simply buck basic errors away, but leave the company when in doubt.
• If employees are virtually forced to make estimates, they will make very conservative estimates out of self-protection. This effect can in turn lead to so-called task-bloating: if I have estimated five days for a user story, but am already finished after one day, no one should rely on the fact that the remaining four days will not be “invested” in other, not necessarily company-related goals.

I give a realistic approach to dynamic project planning little chance. People seek security, want to maintain the familiar at all costs. Illusions are often preferred to an honest admission of ignorance. In the case of software projects, this is demonstrated by the fact that people prefer to work with the illusion of static and plannable costs rather than accepting that, in the case of individual solutions, no static figure can be given as a cost factor other than a rough estimate.

All parties involved have to learn and understand that the previous concept of static planning does not help anyone in the end, but only leads to massive additional costs.

No matter if you are a Product Manager, Tech Lead, CTO/CPO or in any other role: as a Technical Leader you almost inevitably have to deal with projects and/or product developments and are confronted with this problem. If you manage to establish a working approach based on dynamic planning, you have taken a big step forward, because:

• You continuously involve all affected people and roles in the development (what is often called interdisciplinary teams). In order to stay up to date, status information must be regularly collected and understood by all involved.
• A fast feedback rhythm is thus automatically given, because delays or even faster progress are immediately reflected in the forecast. Everyone involved gets a very early update and the entire team can get together promptly in case of doubt and clarify problems in the schedule with each other.
• You take the pressure off yourself and the team from making false statements about deadlines. Professional product development is based on facts and not on cumulative estimates based on estimated requirements.
• In retrospectives, the team can fully concentrate on productive optimization and elimination of blockers and does not have to deal with system errors in the sense of fighting symptoms.

So you create a sustainable process that ultimately enables a timely and fact-based environment in which agile – i.e. fast and easy – changes can be reacted to. You live agility, but don’t get caught up in processes and procedures defined by the so-called Agile Frameworks. Keep in mind, however, that for such a way of working, the technical prerequisites must also be in place. In particular, a high degree of automation for deployments and testing are also among the cornerstones of a modern and agile product development team.

What if we leave out the user stories?

Those who think quite radically might also legitimately question the concept of user stories themselves. If the entire development is seen as a continuous and interactive process, it is not absolutely necessary to write user stories. What you should consider, however, are the following points:

• User Stories allow you to use acceptance criteria to drive compliance with the MVP criteria. If you leave them out, you should have a working concept to prevent over-engineering.
• User stories document requirements engineering. Depending on the industry (health or fintech) and regulatory requirements (ISO 27001 for example), it may be mandatory to document this process. Stories can make the required requirements traceable with all relevant attributes and avoid you having to build and maintain a parallel universe of specifications.
• User stories in a Kanban board can provide an easy-to-understand overview for other people in the company in non-technical roles to understand which functionalities have already been implemented or are yet to come. You should also ensure this transparency in a method without user stories.

If you consider these aspects and find a suitable concept, there is nothing to stop you from establishing an extremely reduced approach without user stories.

Conclusion

Product development in IT is complex. That has always been the case and will remain so. What should not always remain so is the way we deal with this complexity. The often very progressive IT industry in particular should urgently throw some antiquated thought patterns overboard and enhance its reputation and reliability (there is a reason why there is a plethora of more or less tasteless jokes about the duration of projects in IT).

As so often in life, less is sometimes more. Instead of always introducing new so-called agile frameworks, companies should rely on simple methods that are based on real facts and figures. These can give everyone involved an understandable insight into the current status of a project and enable a principle of agile software development: to create added value for users as quickly as possible while working as cost-optimized and efficiently as possible from a business perspective.

However, one thing must be clear: the change to a lean process without story points or estimations and with a burn-up chart or similar tools as visualization is the easy part. The far more difficult part, as always, is the people we need to bring with us. We have a long way to go, away from estimation-based exaggerated and unrealistic expectations of looking into the future and towards a realistic approach based on data and trends for a working collaboration between all people and roles involved.